Ideal OS: Rebooting the Desktop Operating System (2017)(joshondesign.com)
joshondesign.com
Ideal OS: Rebooting the Desktop Operating System (2017)
https://joshondesign.com/2017/08/18/idealos_essay
77 comments
Original author here. I have no idea how my 7 year old post made it to the front page today, but I’m happy to say that the situation is even worse than it was back then. Desktop OSes are even more bloated and less under user control since then.
An improved rant would be nice. I feel the webpage doesn't make enough effort to store the many valuable thoughts and idea the many readers have while reading it. Long ago I had a site that was technically a forum but the front page was an article rather than a table and the sections pointed to sub forums that were articles as well. The later pointed at the usual forum tables. I used the discussion/comments to improve the articles but it should really have been a wiki format.
The text reminded me of the way FTP clients pop up the default text editor. Such a boring technology, no one thought anything of it. Why isn't my mail client doing this? It doesn't seem hard to make web browsers do it either. Trying to do code highlighting in a <textarea> is possibly the worse technology we have. We should really have a dedicated client for forms as well which seems to logically fold out of a database file system?
The text also reminded me of Picassa the photo organizer, as soon as I had everything scanned and organized in it it felt like I couldn't use the normal file system anymore. It was such a clumsy clunker by comparison.
Also hilarious was having to install Thunar because Nautilus couldn't open my big folders.
I was also reminded of textpad. While Atom couldn't open files larger than 2mb the old textpad on a pentium 4 opened the 1+ TB wikipedia xml dump in 0 seconds. Editing was not sluggish, replace macros were not sluggish and saving the document worked just fine. The thought stuck with me: How the f did it just do that??? I think the first release in 1992 probably explains it. Squeezing wine from a rock was still a thing back then.
The text reminded me of the way FTP clients pop up the default text editor. Such a boring technology, no one thought anything of it. Why isn't my mail client doing this? It doesn't seem hard to make web browsers do it either. Trying to do code highlighting in a <textarea> is possibly the worse technology we have. We should really have a dedicated client for forms as well which seems to logically fold out of a database file system?
The text also reminded me of Picassa the photo organizer, as soon as I had everything scanned and organized in it it felt like I couldn't use the normal file system anymore. It was such a clumsy clunker by comparison.
Also hilarious was having to install Thunar because Nautilus couldn't open my big folders.
I was also reminded of textpad. While Atom couldn't open files larger than 2mb the old textpad on a pentium 4 opened the 1+ TB wikipedia xml dump in 0 seconds. Editing was not sluggish, replace macros were not sluggish and saving the document worked just fine. The thought stuck with me: How the f did it just do that??? I think the first release in 1992 probably explains it. Squeezing wine from a rock was still a thing back then.
Was there ever any effort to address these issues that came of your original post?
The state of desktop paradigms has been one of great stagnation, if not outright regression. The VR fad is never going anywhere: its problems are an order of magnitude worse and engineers and companies nowadays seem unable to do basic 2D interface right.
I agree. In my opinion, the problem is that Microsoft and Apple discovered more lucrative markets, making their desktops less important. For Apple, the iPhone’s revenue completely dwarfs the Mac’s, and even services bring in more revenue than the Mac. Microsoft still makes a lot of money selling Windows licenses to OEMs and corporations, but somehow Microsoft still feels compelled to extract even more money from Windows by placing ads in the OS and by selling subscription services.
As a fan of desktop GUI software that conforms to well-developed platform UI guidelines such as the old Apple Human Interface Guidelines, I believe the rise of mobile computing and the Web hurt the desktop. Software companies ship cross-platform applications using least-common-denominator frameworks instead of writing apps with UIs that are tailor-made to each platform. While this saves development time, and while it’s popular to say “having an Electron app is better than having no app,” it makes the desktop experience less consistent and more difficult to use. In addition, the desktop UI has become more mobile-like even in ways that are inappropriate for desktop computing, such as the proliferation of hamburger windows, the tendency to hide settings (thus requiring additional clicks to perform operations that used to be done in one or two clicks), oversized bars, title bars being crammed with buttons (making it harder to move windows via dragging the title bar), excessive padding, etc. There is a difference between making things easier for users versus dumbing things down, but I feel that a lot of software designers these days don’t know the difference.
I would like to see a return to UI consistency across applications, and I’d also like to see a desktop environment that caters to power users.
A dream of mine is some type of “component-based” desktop operating system, where everything is an object that can be manipulated through either a GUI, the command line, or programmatically. The backend would be inspired by technologies like Smalltalk, GNU Emacs, Lisp machines, and Apple OpenDoc, but with modern considerations such as security and parallelism.
As a fan of desktop GUI software that conforms to well-developed platform UI guidelines such as the old Apple Human Interface Guidelines, I believe the rise of mobile computing and the Web hurt the desktop. Software companies ship cross-platform applications using least-common-denominator frameworks instead of writing apps with UIs that are tailor-made to each platform. While this saves development time, and while it’s popular to say “having an Electron app is better than having no app,” it makes the desktop experience less consistent and more difficult to use. In addition, the desktop UI has become more mobile-like even in ways that are inappropriate for desktop computing, such as the proliferation of hamburger windows, the tendency to hide settings (thus requiring additional clicks to perform operations that used to be done in one or two clicks), oversized bars, title bars being crammed with buttons (making it harder to move windows via dragging the title bar), excessive padding, etc. There is a difference between making things easier for users versus dumbing things down, but I feel that a lot of software designers these days don’t know the difference.
I would like to see a return to UI consistency across applications, and I’d also like to see a desktop environment that caters to power users.
A dream of mine is some type of “component-based” desktop operating system, where everything is an object that can be manipulated through either a GUI, the command line, or programmatically. The backend would be inspired by technologies like Smalltalk, GNU Emacs, Lisp machines, and Apple OpenDoc, but with modern considerations such as security and parallelism.
A lot of blame gets placed on mobile, but the problem predates it. Windows being the leader refused to implement any notable standards for decades and still doesn't. This led to standards developed for mobile getting applied elsewhere, just because there's no other standard really and Microsoft's official stance is just to use those same mobile standards anyways, and to allow developers to do whatever they want anyways.
Standardizing on an established shared standard was never motivating for Microsoft. Early they profited by ensuring backwards compatibility for ridiculous periods of time. That was their pitch to customers. Buy your software and it will likely still work in 10 years. Even when shifting from dos pre NT windows you had an OS that allowed programs that thought they had sole control of the machine to continue to run by using some fairly sophisticated techniques and features introduced with the 386. Their model: get users, keep users, get recurring profit was their model. Standardizing on anything didn't really matter unless a significant user base wanted it.
And not having open or shared standards was viewed as a feature to drive down competition.
ATT started out UNIX with their hands tied. Hence BSD was birthed and then later ATT tried to put the genie back in the bottle. But it the economics were different. Hardware costs were the main driver. The OS was a given.
Starting over sounds great. But if history repeats itself, which it usually does, then we are far more likely to see something take (copy) ideas from current OSes and evolve them rather than a radical sea change where we star over.
Also now that Microsoft is shifting (shifted?) to a business model where selling OSws isn't really profitable will their behavior change?
And not having open or shared standards was viewed as a feature to drive down competition.
ATT started out UNIX with their hands tied. Hence BSD was birthed and then later ATT tried to put the genie back in the bottle. But it the economics were different. Hardware costs were the main driver. The OS was a given.
Starting over sounds great. But if history repeats itself, which it usually does, then we are far more likely to see something take (copy) ideas from current OSes and evolve them rather than a radical sea change where we star over.
Also now that Microsoft is shifting (shifted?) to a business model where selling OSws isn't really profitable will their behavior change?
I really notice this when I play PC games. They are easy to get into, they are highly customizable, and they teach you bit by bit until you're pretty powerful at doing things.
Each game though is super narrow, and is built on OS primitives already established. An OS must be all things to all users and all devs, so some complexity and bloat are inevitable.
Users don't want to start from scratch to learn every program they need. Also, they often bounce off of games with controls outside their comfort zone.
Users don't want to start from scratch to learn every program they need. Also, they often bounce off of games with controls outside their comfort zone.
Phones are stagnating too. I don’t remember anything really innovative coming to my iPhone for at least 5 years. It’s just slow evolution. Hope phones won’t go the path of desktop where it seems they are making them worse with every release.
> Hope phones won’t go the path of desktop where it seems they are making them worse with every release.
Got some bad news for ya...
Got some bad news for ya...
However at least Apple and Google are trying to use something else other than plain C and C++, with proper full stack frameworks.
Microsoft keeps failing at making .NET more relevant across the whole OS stack, as WinDev keeps pushing for C++ and COM, no matter what, and did a big mess of how UWP could have improved the whole experience, after the Longhorn sabotage from WinDev.
Microsoft keeps failing at making .NET more relevant across the whole OS stack, as WinDev keeps pushing for C++ and COM, no matter what, and did a big mess of how UWP could have improved the whole experience, after the Longhorn sabotage from WinDev.
It's interesting to me that this article was written ~seven years ago and already feels thoroughly outdated. The core problem is that its basic premise is that we need to scratch-write an OS that is essentially the Alto WIMP (1972), a few BeOS features (1995), and questionable improvements like forcing apps through a messaging bus a la Smalltalk (1972) or CORBA (1991).
It isn't that the individual ideas are bad, per se, but if you're going to suggest write an OS from scratch then it really should be significantly different and inherently impossible to make by tweaking an existing OS, of which there are so, so many. Just an unbelievable number of OS options.
Are there really so many OS options these days? Everything is pretty much "a Unix" of some sort, or else Windows.
There is a key difference between today and the period of the 80s/90s when we had true OS efflorescence. Back then, the real concern was compatibility of software and in particular file/data formats. These days we have so many data format standards and protocols that a lot of this interoperability is essentially system-agnostic. I think that's a huge advantage.
There is a key difference between today and the period of the 80s/90s when we had true OS efflorescence. Back then, the real concern was compatibility of software and in particular file/data formats. These days we have so many data format standards and protocols that a lot of this interoperability is essentially system-agnostic. I think that's a huge advantage.
It's easy to make something lean when you can start from scratch. Backwards compatibility and large user bases resistant to change are overlooked as primary causal factors of the modern state of affairs, imo.
I understood the ideia and agree with it.
Just want to point out that starting an OS from scratch, even if reusing an existing kernel, is a monstrous epic.
I find it difficult that one would be able to just use existing kernels like Linux without some considerable changes to support a more secure userspace (which seems like a requirement in order to do wild new things).
Just want to point out that starting an OS from scratch, even if reusing an existing kernel, is a monstrous epic.
I find it difficult that one would be able to just use existing kernels like Linux without some considerable changes to support a more secure userspace (which seems like a requirement in order to do wild new things).
I wrote a response but it is too long (11019 bytes is too long, apparently). (I have saved a copy on my computer, so that I may post it later (fully or partially).)
But, I have commented about most of the article, and about my own ideas of operating system design (which is capability-based); there are some similarities, but many things are done differently. However, some of the functionality they describe would be possible with how my ideas are doing, too.
But, I have commented about most of the article, and about my own ideas of operating system design (which is capability-based); there are some similarities, but many things are done differently. However, some of the functionality they describe would be possible with how my ideas are doing, too.
Sounds like you should make a blog post!
Actually I had done: scorpion://zzo38computer.org/blog/c_OPERATING_SYSTEM_DESIGN.scorp (I may write more later, too)
What format and URL scheme is that? I've never seen this before in 39 years on the Internet.
It is a more recent one (made up from criticism of Gemini protocol/file-format, as well as criticism of WWW).
(Simple explanation: Use port 1517. Send R and then a space and then the full URL and then carriage return and line feed.)
The file format is a binary format, although there is also a plain text file describing the protocol and file format at: scorpion://zzo38computer.org/specification.txt
(Simple explanation: Use port 1517. Send R and then a space and then the full URL and then carriage return and line feed.)
The file format is a binary format, although there is also a plain text file describing the protocol and file format at: scorpion://zzo38computer.org/specification.txt
This is the opposite of helpful.
What can connect to it? How do I use it? If I need to implement my own protocol, then the answer is "no" followed by "what are you on?"
What can connect to it? How do I use it? If I need to implement my own protocol, then the answer is "no" followed by "what are you on?"
I wrote a "astroget" program which can download the file (and supports several protocols, including Gemini, Gopher, Spartan, HTTP, etc). It is also possible to use netcat. The program to display the file is not yet written, as far as I know. However, the specification file is plain text file, in order to ensure that you can read it without needing to implement it at first.
Hint: don't pull foolish stunts like this, unless you are trying to annoy people.
I’d love to hear about it.
Most of my notes about operating system design are in paper, and in disorganized files in computer (some of which have not been updated from my changed ideas),
I have written some on Hacker News as well, such as: https://news.ycombinator.com/item?id=40651272 https://news.ycombinator.com/item?id=40676995
(I had also written some on Usenet, and also on IRC.)
Perhaps I could set up a NNTP for discussion, that work can be made to write the specifications (which can be changed as the actual implementation is made, in case that reveals problems with the specifications), and to discuss implementations.
I have written some on Hacker News as well, such as: https://news.ycombinator.com/item?id=40651272 https://news.ycombinator.com/item?id=40676995
(I had also written some on Usenet, and also on IRC.)
Perhaps I could set up a NNTP for discussion, that work can be made to write the specifications (which can be changed as the actual implementation is made, in case that reveals problems with the specifications), and to discuss implementations.
Now I have set up a NNTP: nntp://zzo38computer.org/un2.org.zzo38computer.osdesign (at the time of this writing, there are no messages there; I may add one later, but anyone else also may do so)
Thunderbird chokes on this.
It works with Lynx. I don't know what is the problem with Thunderbird, but I added an article in case it doesn't work for newsgroups that do not already have an article. If you are getting a specific error message, what it is?
Nearly a week and no reply.
I do not understand your motivations here: inventing new Internet protocols, apparently intentionally misusing terminology, etc.
("I set up a NNTP" makes no more sense than "here, try to connect over brown": NNTP is a protocol and the normal usage is to set up a news server over which one communicates via a news client over NNTP.)
Maybe this is meant to be funny, or sarcastic/ironic, or performance art, or something else; I can't tell. What it is not is helpful, informative, or clear.
I do not understand your motivations here: inventing new Internet protocols, apparently intentionally misusing terminology, etc.
("I set up a NNTP" makes no more sense than "here, try to connect over brown": NNTP is a protocol and the normal usage is to set up a news server over which one communicates via a news client over NNTP.)
Maybe this is meant to be funny, or sarcastic/ironic, or performance art, or something else; I can't tell. What it is not is helpful, informative, or clear.
> Nearly a week and no reply.
I do not know what reply was expected.
> Maybe this is meant to be funny, or sarcastic/ironic, or performance art, or something else; I can't tell. What it is not is helpful, informative, or clear.
It is meant to be helpful, informative, and clear. Unfortunately, that is not always the case.
I do not know what reply was expected.
> Maybe this is meant to be funny, or sarcastic/ironic, or performance art, or something else; I can't tell. What it is not is helpful, informative, or clear.
It is meant to be helpful, informative, and clear. Unfortunately, that is not always the case.
> It is meant to be helpful, informative, and clear.
It is none of these. Please try harder.
I was interested in following these links and reading about this, but your strange attitude, opaque and incomprehensibly cryptic responses, and general lack of cooperation mean that I have now lost interest.
If you want to play silly games, invent your own protocols and your own unique way of misusing existing language, post "links" that only work using your own client software, and other strange lateral responses, that is of course up to you. Perhaps it is some sort of act.
But do not expect everyone to like it or to be amused. I am not. I am just annoyed. I suspect most people won't even give it that much attention.
It is none of these. Please try harder.
I was interested in following these links and reading about this, but your strange attitude, opaque and incomprehensibly cryptic responses, and general lack of cooperation mean that I have now lost interest.
If you want to play silly games, invent your own protocols and your own unique way of misusing existing language, post "links" that only work using your own client software, and other strange lateral responses, that is of course up to you. Perhaps it is some sort of act.
But do not expect everyone to like it or to be amused. I am not. I am just annoyed. I suspect most people won't even give it that much attention.
Since when does Lynx include a Usenet client?
I don't know when that feature was added, but I know it is able to load the URL starting by "nntp:"; I tried it and it works OK.
Is it possible you send me an email?
I could, but I would prefer not to. Public communication would be preferable, such as by using NNTP. (I may set up a NNTP for this; I have a NNTP server set up already and can easily add a newsgroup.)
This is almost making me want to try out my ideas.
https://gavinhoward.com/2024/05/a-new-architecture-for-a-gui...
https://gavinhoward.com/2024/05/a-better-os-design/
https://gavinhoward.com/2024/05/a-new-architecture-for-a-gui...
https://gavinhoward.com/2024/05/a-better-os-design/
Is getting rid of command line interfaces really a good idea?I don't think so,a lot of people work with and are fond of the CLI tool.
If the CLI were to go away, the OS would still need to have rich APIs for automation. While it may not be something the average user ever uses or knows about, it’s vital for power users and the enterprise.
And then you want these automation APIs to be accessible through a REPL for quick prototyping and...
Either that or the OS need to provide an API to allow major languages to be able to interact with the system other than the GUI. sh can be an add on.
Not sure if this is what you meant, but I'd like to expand a little bit on it.
- This API should be programming language agnostic (like, with a well defined ABI). Ideally an IPC based one, with a well defined wire format and protocols.
- Not a singular API, but an “API system” with extensibility taken into account from the beginning (with things like versioning). It should be able to support arbitrary new API definitions so that other apps could consume, and also produce.
- Such an API system should be used by the OS itself too, of course. So the OS would have the resulting benefits and the whole system would be more uniform.
- This API should be programming language agnostic (like, with a well defined ABI). Ideally an IPC based one, with a well defined wire format and protocols.
- Not a singular API, but an “API system” with extensibility taken into account from the beginning (with things like versioning). It should be able to support arbitrary new API definitions so that other apps could consume, and also produce.
- Such an API system should be used by the OS itself too, of course. So the OS would have the resulting benefits and the whole system would be more uniform.
Later in the post he elaborates on this. I think he's more opposed to the pure text aspect of it.
>Command line interfaces don't fit modern application usage. We simply can't do everything with pure text. I'd like to pipe my Skype call to a video analysis service while I'm chatting, but I can't really run a video stream through awk or sed.
>Command line interfaces don't fit modern application usage.
I guess you and I have different use cases and definition of "modern application usage". I assume you spend most of your day in a GUI, while I spend most of mine in a terminal.
Pipes can handle binary data and sure you could pipe Skypes output through sed or awk... Though something like ffmpeg is more likely, and it will happily take stdin as input.
I guess you and I have different use cases and definition of "modern application usage". I assume you spend most of your day in a GUI, while I spend most of mine in a terminal.
Pipes can handle binary data and sure you could pipe Skypes output through sed or awk... Though something like ffmpeg is more likely, and it will happily take stdin as input.
I am 99% in the CLI.
I just want eyeball tracking for the mouse and voice to LLM to bash.
I just want eyeball tracking for the mouse and voice to LLM to bash.
Depends entirely on what you replace it with. A fun thought, key bindings and task visualisations could mimic cli
"Essentially every application on my computer is a silo."
So true. And so is every company. They have very little incentives to make your data accessible. Or is it "their" data?
You can also go "turtles all the way down"... and discover: the object oriented programming model. It also hides, eh "encapsulates" your data, you only get access through messages.
This is what operating systems are build on. And it works very well for Microsoft / Apple / Google.
I think Unix had the right idea. Make the data part of the outside and programs only process this data. Programs should be stateless.
Unfortunately Unix kind of got stuck in the text based world. Text is really great, because it's so visible. But there is no reason binary data can't be the same. You'd need to develop general purpose binary file format. That can be opened and edited outside of any application. Which would let you extract for example a single image of a video to edit in a paint app.
Then make all of this live update, so that the mouse for example is just a struct with position and other data you can use everywhere.
Then let us have history on all the data.
So true. And so is every company. They have very little incentives to make your data accessible. Or is it "their" data?
You can also go "turtles all the way down"... and discover: the object oriented programming model. It also hides, eh "encapsulates" your data, you only get access through messages.
This is what operating systems are build on. And it works very well for Microsoft / Apple / Google.
I think Unix had the right idea. Make the data part of the outside and programs only process this data. Programs should be stateless.
Unfortunately Unix kind of got stuck in the text based world. Text is really great, because it's so visible. But there is no reason binary data can't be the same. You'd need to develop general purpose binary file format. That can be opened and edited outside of any application. Which would let you extract for example a single image of a video to edit in a paint app.
Then make all of this live update, so that the mouse for example is just a struct with position and other data you can use everywhere.
Then let us have history on all the data.
Just think about the new kind of OS one could make when the computing system is built entirely of low-level actors. uFork [1] is a promising example, and has clear motivations [2].
[1] https://github.com/organix/uFork
[2] http://www.dalnefre.com/wp/2022/08/memory-safety-simplifies-...
[1] https://github.com/organix/uFork
[2] http://www.dalnefre.com/wp/2022/08/memory-safety-simplifies-...
This is a nice document, and many of the suggestions dovetail with my own ideas for a component-based operating system, which I wrote about in 2020 (http://mmcthrow-musings.blogspot.com/2020/04/a-proposal-for-...) but haven’t had the time to work on. I wonder if there’s been any follow-up work on Ideal OS?
Some of my ideas, also.
I agree with the four items listed in "Problems with Today's Desktop Environments and Applications". However, these are not the only issues. (I even thing the older Windows 3.1 and even older UI were well enough; you don't need such fancy graphics.)
Your "Monolithic Applications versus Composable Tools" also is reasonable. What you list about Unix philsophy are the good ideas of Unix philsophy; it is too bad that many modern programs don't support it. I would allow command-line and GUI and easily interact between them.
My idea involves capability-based security with proxy capabilities, and a program can send/receive/wait with capabilities it has access to; a message send/received consists of bytes and/or capabilities. (It is also similar than actor model in some ways.) This is what the kernel does (and programs can create their own capabilities called "proxy capabilities"); all I/O is required to use capabilities. (When a program starts, it must receive an initial message containing some capabilities, so that it can do I/O.)
My idea also is to make the format which will be called "Common Data Format"; it can store documents, databases, and other stuff, and can be used in command-line and GUI. (It also involves some of the ideas of OpenDoc, and of TRON Application Databus, but is different from both of them.)
There are also many other considerations: hypertext file system, window indicators, security, TRON character code, network transparency, emulation, etc.
About "Not a New Operating System", my idea would be that the specification can be done and implementations of the individual parts can be done independently and interchangeable, and that an implementation can be made to run stand-alone or inside of another operating system (something that Inferno also does); this can avoid the problems of drivers at first, while when it is made stand-alone, drivers can be made for it in that way, too. However, I do not intend to include a web browser by default.
(For making such a thing on Linux, I don't know exactly what capsicum blocks; capabilities are supposed to required for all I/O including accessing the current date/time. My system idea is similar like if you have only socketpair to create new channels of communication, and being able to send/receive messages that may contain SCM_RIGHTS, and the ability to wait for objects.)
I agree with the four items listed in "Problems with Today's Desktop Environments and Applications". However, these are not the only issues. (I even thing the older Windows 3.1 and even older UI were well enough; you don't need such fancy graphics.)
Your "Monolithic Applications versus Composable Tools" also is reasonable. What you list about Unix philsophy are the good ideas of Unix philsophy; it is too bad that many modern programs don't support it. I would allow command-line and GUI and easily interact between them.
My idea involves capability-based security with proxy capabilities, and a program can send/receive/wait with capabilities it has access to; a message send/received consists of bytes and/or capabilities. (It is also similar than actor model in some ways.) This is what the kernel does (and programs can create their own capabilities called "proxy capabilities"); all I/O is required to use capabilities. (When a program starts, it must receive an initial message containing some capabilities, so that it can do I/O.)
My idea also is to make the format which will be called "Common Data Format"; it can store documents, databases, and other stuff, and can be used in command-line and GUI. (It also involves some of the ideas of OpenDoc, and of TRON Application Databus, but is different from both of them.)
There are also many other considerations: hypertext file system, window indicators, security, TRON character code, network transparency, emulation, etc.
About "Not a New Operating System", my idea would be that the specification can be done and implementations of the individual parts can be done independently and interchangeable, and that an implementation can be made to run stand-alone or inside of another operating system (something that Inferno also does); this can avoid the problems of drivers at first, while when it is made stand-alone, drivers can be made for it in that way, too. However, I do not intend to include a web browser by default.
(For making such a thing on Linux, I don't know exactly what capsicum blocks; capabilities are supposed to required for all I/O including accessing the current date/time. My system idea is similar like if you have only socketpair to create new channels of communication, and being able to send/receive messages that may contain SCM_RIGHTS, and the ability to wait for objects.)
Every time I boot up Squeak, I think about what the desktop os could have been had all the millions of dollars and hours of work gone in that direction instead
Why not just build your OS on top of Linux? Then you can get driver support at least.
Because Linux, the kernel, as well as other Unixes, don't have the right abstractions to easily build these. And there's all the old cruft.
Just look at what Android had to do to implement a secure platform with somewhat of a well defined non-1980's OS API - it basically removed the Unix part of the OS and built a Microkernel-like thing on top of it.
Driver support would be a very important part though.
Just look at what Android had to do to implement a secure platform with somewhat of a well defined non-1980's OS API - it basically removed the Unix part of the OS and built a Microkernel-like thing on top of it.
Driver support would be a very important part though.
> Just look at what Android had to do to implement a secure platform with somewhat of a well defined non-1980's OS API - it basically removed the Unix part of the OS and built a Microkernel-like thing on top of it.
...Huh? Pretty much all Android adds to the kernel is Binder, which is really a performance optimization for it's version of IPC, which today has equivalents that mean it probably wouldn't need it's own version now if it didn't need compatibility.
The security model allows native apps, which have access to all the normal libc, or even direct syscalls. Some hardware access is marshalled through userspace daemons, which may implement security for things like camera/microphone access, but that's been a pretty common thing on other unix-style OSs for ages.
Though the separation of each app into it's own UID might not have been the original intent, but it seems to work implemented with those same unix capabilities.
Yeah, most apps probably don't use POSIX APIs directly, and instead the java Android APIs, but they are very much build on top of the "crufty 1980s OS APIs", rather than bypassing them.
...Huh? Pretty much all Android adds to the kernel is Binder, which is really a performance optimization for it's version of IPC, which today has equivalents that mean it probably wouldn't need it's own version now if it didn't need compatibility.
The security model allows native apps, which have access to all the normal libc, or even direct syscalls. Some hardware access is marshalled through userspace daemons, which may implement security for things like camera/microphone access, but that's been a pretty common thing on other unix-style OSs for ages.
Though the separation of each app into it's own UID might not have been the original intent, but it seems to work implemented with those same unix capabilities.
Yeah, most apps probably don't use POSIX APIs directly, and instead the java Android APIs, but they are very much build on top of the "crufty 1980s OS APIs", rather than bypassing them.
Android blocks most of Linux syscalls with seccomp. They make heavy usage of SELinux.
Binder is not just a simple IPC system. It requires userspace parts which aren't even the parties who are talking through IPC. Like you said it has lots of performance optimizations, and that requires it hooking up to a number of kernel subsystems. At the beginning it was met with a lot of resistance from kernel maintarners when tried to be merged.
Why did Google came up with it? When talking about the 'Android sandbox' people usually talk about how Android gives each app an different uid/gid. But that's just the beginning of the story. It wasn't enough for sandboxing needs since the beginning. Thats why they took the binder idea from BeOS and used on Android.
Since then Google has blocked everything. Native code is needed on any practical system, but today's native code has access mostly to nothing, just the common needs to run (like libc which is used due backward compatibility). The proper way to get something is through Binder (and even that is fine-grained controlled on a per process basis).
Linux and Unixes in general don't implement true capabilities. Android does through Binder, but since Binder lives “inside Linux” everything else has to be blocked to make these capabilities useful.
> Some hardware access is marshalled through userspace daemons
Most of stuff are, at least initially, orchestrated through userspace software - daemons like you said, but mostly HALs (which are also daemons).
The main takeaway is that binder is _the core_ of Android userspace. It's tightly integrated with SELinux and everything passes through it at some point. Permissions are enforced through it. In the end, the whole “binder subsystem” is architectured like a Microkernel system.
Binder is not just a simple IPC system. It requires userspace parts which aren't even the parties who are talking through IPC. Like you said it has lots of performance optimizations, and that requires it hooking up to a number of kernel subsystems. At the beginning it was met with a lot of resistance from kernel maintarners when tried to be merged.
Why did Google came up with it? When talking about the 'Android sandbox' people usually talk about how Android gives each app an different uid/gid. But that's just the beginning of the story. It wasn't enough for sandboxing needs since the beginning. Thats why they took the binder idea from BeOS and used on Android.
Since then Google has blocked everything. Native code is needed on any practical system, but today's native code has access mostly to nothing, just the common needs to run (like libc which is used due backward compatibility). The proper way to get something is through Binder (and even that is fine-grained controlled on a per process basis).
Linux and Unixes in general don't implement true capabilities. Android does through Binder, but since Binder lives “inside Linux” everything else has to be blocked to make these capabilities useful.
> Some hardware access is marshalled through userspace daemons
Most of stuff are, at least initially, orchestrated through userspace software - daemons like you said, but mostly HALs (which are also daemons).
The main takeaway is that binder is _the core_ of Android userspace. It's tightly integrated with SELinux and everything passes through it at some point. Permissions are enforced through it. In the end, the whole “binder subsystem” is architectured like a Microkernel system.
I'd argue that permissions aren't really enforced by binder, just binder is used to talk to the endpoints that enforce permissions. Binder has no interaction with SELinux outside the policy requirements to allow apps to open it's device nodes - there's no interaction in the kernel code at all [0]. It's just an opaque data blob IPC mechanism, not tagging payloads with capabilities or similar in-kernel.
At the point that binder was created, there wasn't a good kernel-level IPC system integrated into linux, true, but I'm not sure I agree with the idea that it's in any way tied into the capabilities/permissions system outside from being a communication channel to the userspace daemons things that do.
And native code does have access to every service on the device - just as any app. It's just more painful than the Java layer, and some things aren't guaranteed to be stable over API versions. Less useful for apps, but still not part of the security model. At an extreme level, the java code is run in the same memory address space as any native code, there's no protection from that. Hell, some apps even used to patch dalvik/ART live. It was insane and fragile, but entirely possible.
And the "disallowed" syscalls are those not exposed by libc though some more are restricted for user apps (mostly around UID management, which is the big "major" deviation from standard unix, and system stuff like loading kernel modules or rebooting...) [1] So maybe it doesn't allow some linux extensions, or could remove the "old" versions of replaced syscalls (when arguments were found not to be large enough, flags arguments added etc.). But there's not really anything missing from there as used in 99% of unix userspace programs anyway.
So I maintain it's still a system build on top of a unix kernel and libc with a few additions on the side, and all that is still accessible to apps. Perhaps you think I'm including all the stuff people associate with modern "Desktop Linux"? Like X11 and/or wayland? Or the entire FHS? That's the main thing Android has completely divorced from, but that also isn't "Unix", or even really from the 80s even if it has some ancestors there.
[0] https://android.googlesource.com/kernel/common/+/refs/heads/...
[1] See SECCOMP*.txt and SYSCALLS.txt from https://android.googlesource.com/platform/bionic/+/refs/head...
At the point that binder was created, there wasn't a good kernel-level IPC system integrated into linux, true, but I'm not sure I agree with the idea that it's in any way tied into the capabilities/permissions system outside from being a communication channel to the userspace daemons things that do.
And native code does have access to every service on the device - just as any app. It's just more painful than the Java layer, and some things aren't guaranteed to be stable over API versions. Less useful for apps, but still not part of the security model. At an extreme level, the java code is run in the same memory address space as any native code, there's no protection from that. Hell, some apps even used to patch dalvik/ART live. It was insane and fragile, but entirely possible.
And the "disallowed" syscalls are those not exposed by libc though some more are restricted for user apps (mostly around UID management, which is the big "major" deviation from standard unix, and system stuff like loading kernel modules or rebooting...) [1] So maybe it doesn't allow some linux extensions, or could remove the "old" versions of replaced syscalls (when arguments were found not to be large enough, flags arguments added etc.). But there's not really anything missing from there as used in 99% of unix userspace programs anyway.
So I maintain it's still a system build on top of a unix kernel and libc with a few additions on the side, and all that is still accessible to apps. Perhaps you think I'm including all the stuff people associate with modern "Desktop Linux"? Like X11 and/or wayland? Or the entire FHS? That's the main thing Android has completely divorced from, but that also isn't "Unix", or even really from the 80s even if it has some ancestors there.
[0] https://android.googlesource.com/kernel/common/+/refs/heads/...
[1] See SECCOMP*.txt and SYSCALLS.txt from https://android.googlesource.com/platform/bionic/+/refs/head...
> I'd argue that permissions aren't really enforced by binder, just binder is used to talk to the endpoints that enforce permissions. Binder has no interaction with SELinux outside the policy requirements to allow apps to open it's device nodes - there's no interaction in the kernel code at all [0]. It's just an opaque data blob IPC mechanism, not tagging payloads with capabilities or similar in-kernel.
I'm going to describe some flows so you can have a perspective on what's going on. I'm describing them as I remember when I worked with this stuff.
First, you have to know about SEAndroid which is a thing built on top of SELinux. It doesn't add anything new from a kernel perspective but there's a whole 'firmware image build-time framework' to _legislate_/declare permissions about all sensitive stuff on the system. Like, you want to add a new daemon to the system? Maybe this daemon will serve some AIDL (such daemon is called a service)? Oh, then you should declare its SELinux stuff, like "it's a daemon", "it should have access to sockets/whatever" and "it will serve such and such AIDL interfaces" and so on. You should state the things it should have access to. The important part is that the SEAndroid stuff has 'Binder services' as a first class citizen, much like a fs path, socket or anything else SELinux is capable of legislating about (even though the kernel and kernel's binder knows nothing about binder services from an SELinux perspective). A good part of these policies don't talk about vanilla Linux stuff (like processes, files, sockets, etc.) but are about _services_ from "binder realm" - the kernel doesn't know about and ignores them. During the system image build process these policies are "compiled" and put somewhere on a filesystem.
When Android boots, after all the device-mapper mess (which, as most things, is a subject on its own) eventually the "base" filesystems are mounted (this mess comes in part from Project Treble). At this point there's no SELinux stuff loaded (which means, IIRC the SELinux status from the kernel is "off"/insecure/whatever). Then comes the time when the SEAndroid/SELinux policies are to be loaded. A system process looks up and loads the SELinux blobs generated during the system's image build process, and then enables SELinux.
Ok, now a flow from runtime:
You have process A, which in this example will be on the client role, and process B which will be the server. Process A comes from a program A which is a plain ELF binary on the filesystem. Same for process B. The SEAndroid policies loaded during boot define the SELinux contexts associated with process A and B. Program's B context includes the list of AIDL interfaces/services it implements (or serves). And program's A context includes an statement that it should have access to such and such AIDL services (it states nothing about program B or others who implement the AIDL services).
Every binder driver device is exposed to userspace through the /dev fs and they each need to have an associated _service manager_ process. When I worked with these there were 3 binder devices, and so 3 associated service managers. Originially there was just 1 binder. So process A needs to have access to, say /dev/binder, and this is enforced with vanilla Linux acccess controls. Same for process B on this part. Process A tries to establish communication with whatever process is implementing the AIDL service it's trying to talk to (process B) (services have a string name, it's a binder thing). This attempt actually ends up on the binder's service manager, which asks the kernel's SELinux policy 'Can this process A here access that AIDL interface?'. The kernel doesn't know about AIDL interfaces, but the SELinux policy blobs downloaded into it earlier encode this info in such a way that such questions can be asked and answered. If so, it awakes/starts process B and communication goes on.
As an addendum, binder supports transporting file descriptors and that's what I meant by true capabilities. That has lots of implications and caveats given it's going on a Linux-based system but that's another conversation...
> And native code does have access to every service on the device
No, it dependes on which process, or more specifically, which SELinux context the process running that code has been attributed to.
> At an extreme level, the java code is run in the same memory address space as any native code, there's no protection from that
Yes, on the level this conversation is concerned about, it doesn't matter what you program is doing, if it's a Java thing, python, web browser, if it's a code dynamically linked into your process from a shared library object - what matters most is that it ends up calling syscalls and that each process has an identity from the SELinux perspective inside the kernel (so the SELinux subsystem inside the kernel knows which process is calling a given syscall).
> Perhaps you think I'm including all the stuff people associate with modern "Desktop Linux"? Like X11 and/or wayland? Or the entire FHS?
My comment about this part would be that libc is... not related at all. It's importance comes from 'Oh, this is C/C++ code so it doesn't actually issue syscalls, it does all that though a libc because, well, this is C and Unix'. On this abstraction level I would put it as just an implementation detail as apps being programmed in Java.
I'm going to describe some flows so you can have a perspective on what's going on. I'm describing them as I remember when I worked with this stuff.
First, you have to know about SEAndroid which is a thing built on top of SELinux. It doesn't add anything new from a kernel perspective but there's a whole 'firmware image build-time framework' to _legislate_/declare permissions about all sensitive stuff on the system. Like, you want to add a new daemon to the system? Maybe this daemon will serve some AIDL (such daemon is called a service)? Oh, then you should declare its SELinux stuff, like "it's a daemon", "it should have access to sockets/whatever" and "it will serve such and such AIDL interfaces" and so on. You should state the things it should have access to. The important part is that the SEAndroid stuff has 'Binder services' as a first class citizen, much like a fs path, socket or anything else SELinux is capable of legislating about (even though the kernel and kernel's binder knows nothing about binder services from an SELinux perspective). A good part of these policies don't talk about vanilla Linux stuff (like processes, files, sockets, etc.) but are about _services_ from "binder realm" - the kernel doesn't know about and ignores them. During the system image build process these policies are "compiled" and put somewhere on a filesystem.
When Android boots, after all the device-mapper mess (which, as most things, is a subject on its own) eventually the "base" filesystems are mounted (this mess comes in part from Project Treble). At this point there's no SELinux stuff loaded (which means, IIRC the SELinux status from the kernel is "off"/insecure/whatever). Then comes the time when the SEAndroid/SELinux policies are to be loaded. A system process looks up and loads the SELinux blobs generated during the system's image build process, and then enables SELinux.
Ok, now a flow from runtime:
You have process A, which in this example will be on the client role, and process B which will be the server. Process A comes from a program A which is a plain ELF binary on the filesystem. Same for process B. The SEAndroid policies loaded during boot define the SELinux contexts associated with process A and B. Program's B context includes the list of AIDL interfaces/services it implements (or serves). And program's A context includes an statement that it should have access to such and such AIDL services (it states nothing about program B or others who implement the AIDL services).
Every binder driver device is exposed to userspace through the /dev fs and they each need to have an associated _service manager_ process. When I worked with these there were 3 binder devices, and so 3 associated service managers. Originially there was just 1 binder. So process A needs to have access to, say /dev/binder, and this is enforced with vanilla Linux acccess controls. Same for process B on this part. Process A tries to establish communication with whatever process is implementing the AIDL service it's trying to talk to (process B) (services have a string name, it's a binder thing). This attempt actually ends up on the binder's service manager, which asks the kernel's SELinux policy 'Can this process A here access that AIDL interface?'. The kernel doesn't know about AIDL interfaces, but the SELinux policy blobs downloaded into it earlier encode this info in such a way that such questions can be asked and answered. If so, it awakes/starts process B and communication goes on.
As an addendum, binder supports transporting file descriptors and that's what I meant by true capabilities. That has lots of implications and caveats given it's going on a Linux-based system but that's another conversation...
> And native code does have access to every service on the device
No, it dependes on which process, or more specifically, which SELinux context the process running that code has been attributed to.
> At an extreme level, the java code is run in the same memory address space as any native code, there's no protection from that
Yes, on the level this conversation is concerned about, it doesn't matter what you program is doing, if it's a Java thing, python, web browser, if it's a code dynamically linked into your process from a shared library object - what matters most is that it ends up calling syscalls and that each process has an identity from the SELinux perspective inside the kernel (so the SELinux subsystem inside the kernel knows which process is calling a given syscall).
> Perhaps you think I'm including all the stuff people associate with modern "Desktop Linux"? Like X11 and/or wayland? Or the entire FHS?
My comment about this part would be that libc is... not related at all. It's importance comes from 'Oh, this is C/C++ code so it doesn't actually issue syscalls, it does all that though a libc because, well, this is C and Unix'. On this abstraction level I would put it as just an implementation detail as apps being programmed in Java.
[deleted]
Eh the system described in the post is mostly about taking away (no filesystem access, no multiple IPC methods, no X11...) All new required software seems to be only userspace:
- New IPC system: presumably it's abstracted, so can use any existing IPC for implementation. Or, purely as optimization, use Android's Binder or bring back bus1 (it was such a great idea)
- Document Database: pure userspace; in the most extreme case use raw block device
- Compositor / WM / Windows: this was always userspace already
- Rebuild the apps: userspace as well
Then once you new fancy APIs are ready, you use seccomp and/or per-app users and/or namespaces to isolate apps from everything except the few things people want.
- New IPC system: presumably it's abstracted, so can use any existing IPC for implementation. Or, purely as optimization, use Android's Binder or bring back bus1 (it was such a great idea)
- Document Database: pure userspace; in the most extreme case use raw block device
- Compositor / WM / Windows: this was always userspace already
- Rebuild the apps: userspace as well
Then once you new fancy APIs are ready, you use seccomp and/or per-app users and/or namespaces to isolate apps from everything except the few things people want.
> the system described in the post is mostly about taking away
I agree.
The hard part is, as usual, on the details. How do you take away stuff so that you still end up with an usable general purpose OS? That's definitely not easy, specially if you want to make it in a proper way, without hacking things up.
For example, suppose you make the decision to use the Linux kernel, which is a good one - you get driver support, networking, filesystems, and all the good stuff and general primitives for a common OS of today.. Given that what one wants is to remove stuff, in order to make that secure (which is requirement) you'd need to isolate and block everything because on Linux everything is accessible to a process by default, and the common primitives for control are very basic - you'd have to do a _lot_ of work in userspace to block it. It would contaminate your userspace architecture with details and stuff. And then you'd probably end up needing something to make the userspace parts able to communicate with each other, like Binder on Androids case (see my other comment in this thread).
I agree.
The hard part is, as usual, on the details. How do you take away stuff so that you still end up with an usable general purpose OS? That's definitely not easy, specially if you want to make it in a proper way, without hacking things up.
For example, suppose you make the decision to use the Linux kernel, which is a good one - you get driver support, networking, filesystems, and all the good stuff and general primitives for a common OS of today.. Given that what one wants is to remove stuff, in order to make that secure (which is requirement) you'd need to isolate and block everything because on Linux everything is accessible to a process by default, and the common primitives for control are very basic - you'd have to do a _lot_ of work in userspace to block it. It would contaminate your userspace architecture with details and stuff. And then you'd probably end up needing something to make the userspace parts able to communicate with each other, like Binder on Androids case (see my other comment in this thread).
A lot of this seems like it's done by Arcan (https://arcan-fe.com)
The author's most recent post on this topic is here (2022): https://joshondesign.com/2022/09/21/idealos_mark6
That's interesting -- thanks.
Looks to me like he has some of the same ideas as I had in this talk:
https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-...
I turned it into an article series:
https://www.theregister.com/Tag/One%20Way%20Forward/
Looks to me like he has some of the same ideas as I had in this talk:
https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-...
I turned it into an article series:
https://www.theregister.com/Tag/One%20Way%20Forward/
Very sad article, so much wasted productivity potential...
And it's no better today, unfortunately
> Why can't I have a file in two places at once on my filesystem?
Hard links do that, but as usual the UI around this isn't great
> Why can't I have a file in two places at once on my filesystem?
Hard links do that, but as usual the UI around this isn't great
Only on the same filesystem
you mean this?
> two places at once on my filesystem
> two places at once on my filesystem
Well, it is unclear. In a POSIX-y context "filesystem" can refer to
(a) the entire permanent storage system
(b) a particular technology for organizing bits on a storage system
(c) a subset of (a) that has been mount(1)-ed somewhere within (a)
I tend to use (c) most often. No idea what the author or you might do.
(a) the entire permanent storage system
(b) a particular technology for organizing bits on a storage system
(c) a subset of (a) that has been mount(1)-ed somewhere within (a)
I tend to use (c) most often. No idea what the author or you might do.
If you want to know what I’m currently working on, and that helps in my prototypes: check out https://docs.trunk.io/
Part of me wonders how much of this is hardware vendors releasing unusable hardware. I don't know what the hangup on the RPi is (that seems odd, given how prevalent Linux on the RPi is) but that & the keyboard RGB LEDs seems like your bog standard "the hardware can't or if it can, the vendor didn't bother to document anything". The FOSS world has done an incredible amount of work reverse engineering over the years, but the behavior of HW companies are more to blame, and that's not a problem you'll solve with tech.
But hardware vendors have for years, and still do, just say "our use case, or the highway, you don't really own the thing you're buying." And there's not much out there for one to "vote with one's wallet" on it.
"FS can't store a file in two spots" … I want to accuse OP of being a Windows user, but they're obviously not [only]. So that's … an odd critique?
I still think hierarchical+symlinks beats trying to tag stuff, and I've lived through enough decades of AI trying to tag stuff and deciding things like that Mozart is "Rap [explicit]". I don't have the mental capacity for it. Just give me the dumb hierarchy with (sym)links. (And bind mounts.) (But I wouldn't oppose an OS trying it.)
"Let's ditch all IPC" … buuuut adds IPC right back in. Okay :). I suppose it's "only message passing". (And I do think message passing is a great default, and what most things need.)
"Native apps are bloated". The most bloated apps I have are all the Electron ones. I don't think most people are concluding this...
I'd love a multi-item clipboard, but the devil is in the details: sensitive items and large transfers.
> it's the pure text nature that bothers me. Instead of chaining CLI apps together with text streams we need something richer, like serialized object streams (think JSON but more efficient).
You have to consider the bulk cases. But very much yes; I wish at least that Unix had settled on typed bytestreams, like MIME+stream, and some form of content negotiation. Unix scripts are rife with examples of `a | b` where a & b speak different formats and I feel like it took forever to get good tooling around strongly specified formats (instead of just bullshitting with awk/grep/cut/tr/etc.) with tools like jq, yq, xsv, etc. I just wish applications could just have a means to understand "oh, I'm getting this kind of data" or "I have no idea what this is, and I shouldn't just GIGO".
> In our new OS every window is tab dockable. Or side dockable. Or something else
Have you tried a tiling WM?
If you design an OS ... make the FS Unicode only, and ban "\n" from filenames. Life will be simpler.
But hardware vendors have for years, and still do, just say "our use case, or the highway, you don't really own the thing you're buying." And there's not much out there for one to "vote with one's wallet" on it.
"FS can't store a file in two spots" … I want to accuse OP of being a Windows user, but they're obviously not [only]. So that's … an odd critique?
I still think hierarchical+symlinks beats trying to tag stuff, and I've lived through enough decades of AI trying to tag stuff and deciding things like that Mozart is "Rap [explicit]". I don't have the mental capacity for it. Just give me the dumb hierarchy with (sym)links. (And bind mounts.) (But I wouldn't oppose an OS trying it.)
"Let's ditch all IPC" … buuuut adds IPC right back in. Okay :). I suppose it's "only message passing". (And I do think message passing is a great default, and what most things need.)
"Native apps are bloated". The most bloated apps I have are all the Electron ones. I don't think most people are concluding this...
I'd love a multi-item clipboard, but the devil is in the details: sensitive items and large transfers.
> it's the pure text nature that bothers me. Instead of chaining CLI apps together with text streams we need something richer, like serialized object streams (think JSON but more efficient).
You have to consider the bulk cases. But very much yes; I wish at least that Unix had settled on typed bytestreams, like MIME+stream, and some form of content negotiation. Unix scripts are rife with examples of `a | b` where a & b speak different formats and I feel like it took forever to get good tooling around strongly specified formats (instead of just bullshitting with awk/grep/cut/tr/etc.) with tools like jq, yq, xsv, etc. I just wish applications could just have a means to understand "oh, I'm getting this kind of data" or "I have no idea what this is, and I shouldn't just GIGO".
> In our new OS every window is tab dockable. Or side dockable. Or something else
Have you tried a tiling WM?
If you design an OS ... make the FS Unicode only, and ban "\n" from filenames. Life will be simpler.
It's funny this article is technically correct but also totally wrong: if you can't run the application(web browser, game, office suite) you want on an OS, it maybe the 'Ideal' OS you're still going back to Windows..
There are ways around this in some cases: use emulation, modify the existing program to port it to the new system (if the source code is available), or write a new one. An ideal system will allow all three possibilities; although, which are possible also depends on the specific program that you want to run.
> Twitter user ekse has graciously translated this blog to Chinese.
How far we've come.
How far we've come.
This article predates Apple's visionOS. I wonder what the author's take on it would be.
I love Apple’s hardware and the metaphors are well thought out, but I’m disappointed that one of the best design companies in the world couldn’t come up with better uses than photos and having a virtual Mac monitor.
Because they didn't want it to be another MetaQuest or HoloLens, thus there isn't much left as use cases.