Show HN: The Unite real time operating system(jacquesmattheij.com)
jacquesmattheij.com
Show HN: The Unite real time operating system
https://jacquesmattheij.com/unite-operating-system/
52 comments
That's interesting. Did you have access to the QNX sources at the time and were able to study the QNX kernel and why it had a much better performance than Minix? Given your reference to DJGPP, am I right to assume that you have written your OS in C? (I wasn't able to download the source code and would appreciate to be able to download a compressed file from a "normal" web server). Since you made comparisons with the Minix version of that time, what conclusions did you draw from this, and what especially made your kernel "lightyears ahead" of Minix?
No, but I had used it extensively and written a fairly large amount of code for it so it was mostly re-implementing how I thought things must work under the hood. The hardest part was to bridge some of the posix like functionality to this rather strange set of primitives, especially 'fork' and the '/dev/' parts required some pretty hacky solutions. But they work and they ended up being quite reliable.
As for it being lightyears ahead of Minix: Minix was built from the ground up as a platform to demonstrate how Unix works without being Unix. QnX and by extension my version on that theme was built from day one to be low latency and to have the smallest possible kernel footprint that you could have. The lack of a root file system and the Plan-9 like network transparency are all as a result of that and allow extremely fine grained control over what goes into the final package. It is far more suitable for low level control while at the same time having high level tooling than Minix. What I loved about Minix is the way the source code is organized and how - and this is obviously no accident - it reads as a textbook, I learned a lot from looking at the code and ended up adopting the file system as a stand-alone piece of code (which allowed me to pass files in and out of the OS early on by writing them under one OS and then rebooting so I had access on the other side).
As for it being lightyears ahead of Minix: Minix was built from the ground up as a platform to demonstrate how Unix works without being Unix. QnX and by extension my version on that theme was built from day one to be low latency and to have the smallest possible kernel footprint that you could have. The lack of a root file system and the Plan-9 like network transparency are all as a result of that and allow extremely fine grained control over what goes into the final package. It is far more suitable for low level control while at the same time having high level tooling than Minix. What I loved about Minix is the way the source code is organized and how - and this is obviously no accident - it reads as a textbook, I learned a lot from looking at the code and ended up adopting the file system as a stand-alone piece of code (which allowed me to pass files in and out of the OS early on by writing them under one OS and then rebooting so I had access on the other side).
Amazing, thanks. Is it possible to just have a look at the source code somewhere, e.g. codeberg or gitlab? I'm especially interested in the microkernel IPC.
The send/receive/reply bits are almost trivial, I can mail you the relevant parts. I am explicitly not putting any of this on GitLab or any other medium that is easily ingested by the AI peddlers, if they want my code they'll have to work for it :)
If you drop me a line on [email protected] I'll send you back a zip file.
If you drop me a line on [email protected] I'll send you back a zip file.
In QNX, the (async) IPC is integrated with the scheduler, which is apparently responsible for the performance (still researching). So I was wondering if you chose a similar architecture. My email is "me at rochus-keller dot ch". Thanks.
Yes, that is how it works. Zip file coming up.
Edit: sent. Enjoy!
Edit: sent. Enjoy!
Congrats on shipping Jaques!
Great to see you resurrect this project and publish it.
Great to see you resurrect this project and publish it.
Thank you. One download so far :) Mega success!
It's interesting how you can release an OS + source code and hardly anybody will bat an eye. But nonsense attached to an LLM will draw a crowd ;)
Anyway, I'm happy that I got it to work and I hope at least someone will find it useful.
It's interesting how you can release an OS + source code and hardly anybody will bat an eye. But nonsense attached to an LLM will draw a crowd ;)
Anyway, I'm happy that I got it to work and I hope at least someone will find it useful.
Maybe he should have named the OS the “LLM OS”, and then only later people will realize LLM stands for something else entirely.
That's slightly evil, but funny too!
Just gotta ride the wave as you know.
Sorry I misspelled your name originally!
Sorry I misspelled your name originally!
NP, that's been happening to me pretty much since I was born.
Aw. I've always liked QNX. But a modern implementation, not a revival of a 30 year old abandoned project, would be more useful. We really need something for small IoT boxes that has less attack surface than Linux.
The problem people run into trying to build something like QNX is that, while the kernel is tiny, it won't even do Hello World until you have the necessary user-space drivers running. Initial debugging is hard.
The problem people run into trying to build something like QNX is that, while the kernel is tiny, it won't even do Hello World until you have the necessary user-space drivers running. Initial debugging is hard.
If you're looking for something like QNX you could use... QNX. It's still around and free for hobbyists [0]. You can use it knowing you're using the same OS as your car, or train, or medical device, or spacecraft.
[0] https://www.qnx.com/products/everywhere/
[0] https://www.qnx.com/products/everywhere/
You still have to register with them, I'm not sure that is common among NC licenses. I assume this also means its not really open-source (or compatible with open collaboration) if they are gatekeeping (via licences) the free dev licence.
Also consider this - If you got a licence would this "taint" you if you ever wanted to work on a project like OPs? As in, the registration is proof you could access the proprietary QNX code, forming the basis of a lawsuit.
Also consider this - If you got a licence would this "taint" you if you ever wanted to work on a project like OPs? As in, the registration is proof you could access the proprietary QNX code, forming the basis of a lawsuit.
Quantum has muddied the waters considerably here by at some point releasing the code under a fully open license. They tried hard to reverse that decision but I'm sure there are still copies of that version floating around.
In contrast to Minix 3.0+, no version of QNX has ever been fully open. At its most permissive, it was made available (behind a form submission) under a look-but-don't-touch All Rights Reserved sourceware license.
After previous rugpull, why anyone would want to jump through the hoops?
Yes. The history is awful. Closed source, partially open source with free version, closed source, fully open source with free version (but not for commercial use), and then, suddenly one day, closed source. Twenty years ago, many of the Gnu tools built for QNX by default. That stopped.
You can get a "personal use license" now, but you can't distribute anything that has parts of QNX code in it.
You can get a "personal use license" now, but you can't distribute anything that has parts of QNX code in it.
From what people reported around the rugpull, BB pretty much nuked all customer relationships with various groups that had commercial licenses too, not just with people who looked into open source or free-as-in-beer options.
Greatly accelerated AGL with D-Bus (yuck) as patchwork replacement for QNX IPC
Greatly accelerated AGL with D-Bus (yuck) as patchwork replacement for QNX IPC
Yes, this happened to us as well. It basically killed more than a decade of development. I did write a send/receive/reply/name_attach/name_locate library for Linux, which worked well enough that we could at least rescue the project. But BB killed QnX for large scale software development projects outside of the embedded space, and quite possibly for a lot of those as well (but I had no contact with such groups). There was a point in time where QnX ran a very large fraction of all of the world's infrastructure and BB showed the dangers of relying on a company that never fully committed to their long-term strategy.
Some of the older autonomous challenges involving flight I remember a lot of contestants had QNX on board vs. let's say VxWorks or others.
I also guess QNX having more point-to-point IPC/RPC fits some stuff better without having to plop in (possibly expensive) middleware like CORBA-RT
I also guess QNX having more point-to-point IPC/RPC fits some stuff better without having to plop in (possibly expensive) middleware like CORBA-RT
The way that IPC/RPC worked across the network was the real power of QNX and a very much under appreciated aspect of it. SUN microsystems had this slogan: "The Network Is The Computer". But they never really delivered on that QNX, Erlang and Plan9 actually did.
All of the acquisitions and license issues did QnX no favors. We built a very large installation on top of it and always felt that Quantum didn't really know what they wanted to be. The BB acquisition was the final nail in the coffin, though, as a platform it was a pretty good illustration of how powerful that whole mechanism is.
Yes. The ownership by Harmon (an audio company) was clueless, and Blackberry was, strangely, no better.
I once told one of their marketing guys "Quit worrying about being pirated and worry more about being ignored."
I wonder if a business model like Epic's would have worked. Unreal Engine is free until you hit US$1 million in revenue, and then they want a cut of revenue. This works in games because any game with significant revenue is publicly visible. Less so in operating systems. You could have QNX inside a million traffic lights without anybody knowing.
I once told one of their marketing guys "Quit worrying about being pirated and worry more about being ignored."
I wonder if a business model like Epic's would have worked. Unreal Engine is free until you hit US$1 million in revenue, and then they want a cut of revenue. This works in games because any game with significant revenue is publicly visible. Less so in operating systems. You could have QNX inside a million traffic lights without anybody knowing.
The problem with operating systems is that if they work well they're all but invisible. They only attract attention when they malfunction. QNX allowed us to focus on the problems we were solving, rather than that it drew a lot of attention to itself. I walked into a project that was already years underway and started working on core application components within a few weeks of reading the QNX manual (which I thought was excellent, by the way, not unlike the documentation that came with Mark Williams C). The OS was facilitating what you wanted to do rather than that you were fighting it every step of the way.
The main application we wrote (and of which I rewrote significant chunks) processed a record number of messages on every Monday morning, week after week, year after year. It was a massive cash cow for the company that had commissioned it (KVSA, https://www.kvsa.nl/en/data-and-intelligence/bts-broker-tool... , it is still alive but it no longer runs on QNX) and if not for QNX I doubt it would have gotten as far as it did. Especially in those first few years when it was mostly my old boss and one or two helpers building it. The OS pretty much enforced clean abstraction layers by showing at the OS level itself how this was done. Extending that was a natural path and I suspect that many QNX shops did this simply because it felt right and made it easy. They likely did not realize they were building what we would call a 'service oriented architecture' many years later and in a completely different context.
This experience still leaves me looking at many of the systems we use today as unnecessarily bloated and frankly, quite ugly. QNX was light weight, but not a lightweight. The only other eco-systems that give me that same feeling are Plan9 and Erlang.
The main application we wrote (and of which I rewrote significant chunks) processed a record number of messages on every Monday morning, week after week, year after year. It was a massive cash cow for the company that had commissioned it (KVSA, https://www.kvsa.nl/en/data-and-intelligence/bts-broker-tool... , it is still alive but it no longer runs on QNX) and if not for QNX I doubt it would have gotten as far as it did. Especially in those first few years when it was mostly my old boss and one or two helpers building it. The OS pretty much enforced clean abstraction layers by showing at the OS level itself how this was done. Extending that was a natural path and I suspect that many QNX shops did this simply because it felt right and made it easy. They likely did not realize they were building what we would call a 'service oriented architecture' many years later and in a completely different context.
This experience still leaves me looking at many of the systems we use today as unnecessarily bloated and frankly, quite ugly. QNX was light weight, but not a lightweight. The only other eco-systems that give me that same feeling are Plan9 and Erlang.
"QNX *is light weight, but not a lightweight."
FIFY :)
FIFY :)
QnX is expensive. And in a way that is exactly what I'm hoping for with this release: that someone will pick it up, see how simple a robust OS can be and to use it as a base to build on (for instance, by porting Rust and rewriting the core in that, which would give a very quick path to a self hosting OS), or maybe just as a source of inspiration.
Here you have all of the scaffolding in place already so now you can focus on those other parts one at the time while having the luxury of good tooling and a nice debugging environment, no need to bring it up from scratch. Another great thing would be to make an ARM or other CPU port. And that's not all that hard: there is only one process that is CPU specific and there are a very limited number of pieces that you need (essentially just a COM port driver) to be able to make it self hosting.
Here you have all of the scaffolding in place already so now you can focus on those other parts one at the time while having the luxury of good tooling and a nice debugging environment, no need to bring it up from scratch. Another great thing would be to make an ARM or other CPU port. And that's not all that hard: there is only one process that is CPU specific and there are a very limited number of pieces that you need (essentially just a COM port driver) to be able to make it self hosting.
Back when I did embedded development full-time I always lusted after QNX, but the costs and royalties were higher than what our low-volume product could justify. Or maybe we were just cheapskates.
No, you are right, in volume QNX is expensive. We spent a ton of money ($1600 / license iirc) on licenses for a system that we built, and even so it felt like Quantum never took us serious because we only needed 10's of them rather than the 100's of thousands they were used to selling to their bigger automotive customers. But it is still an OS that will for me always embody the 'road not taken' and I still strongly believe that it was a better road, based on my experience with IRIX, Windows, DOS, VMS, Linux, VAX Unix, OS/9, Plan 9 and QNX. It felt like that only one that always ran and that always stayed responsive, no matter what we threw at it. Across many years and 10's of machines running various versions of QNX I have not seen a single production crash of the OS that was not an indication that the underlying hardware had died. But it was perfectly possible to anticipate even that and have an Erlang like supervisor system in place to deal with that.
> We really need something for small IoT boxes that has less attack surface than Linux.
seL4 exists.
seL4 exists.
People mostly use seL4 as a hypervisor, running Linux on top of it. QNX has a POSIX API, so you don't need another layer.
QNX is allegedly a microkernel. Either the QNX POSIX API is itself another layer over the microkernel, or QNX cannot be a microkernel (hint: it's the former [1]).
People run L4 as a hypervisor only because they want access to the full Linux API. That's not really what you're talking about though, and there's nothing stopping a thin QNX-style POSIX compatibility layer as a user-space server or library as with QNX. The point being, the mature, widely deployed, robust and widely tested microkernel you're looking for already exists, so you just need to focus on the POSIX layer.
[1] https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino...
People run L4 as a hypervisor only because they want access to the full Linux API. That's not really what you're talking about though, and there's nothing stopping a thin QNX-style POSIX compatibility layer as a user-space server or library as with QNX. The point being, the mature, widely deployed, robust and widely tested microkernel you're looking for already exists, so you just need to focus on the POSIX layer.
[1] https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino...
Actually, most of the POSIX compatibility stuff is in QNX's equivalent of "libc". When you call Unix "write", the library does a MsgSend to the appropriate service. MsgSend is in the kernel, but whatever performs the write is in some other process.
Some other functions, such as program loading, are in user space .so files. When you spawn a new process (in QNX, spawn is the usual function, and fork/exec are emulated for backwards compatibility), the library in your program attaches to the .so file that contains the program loader. The user process requests memory and decodes the executable image, and sets up the new process environment. If something in executable file decode does something wrong, the process faults in the usual user space way.
There's no privileged emulation layer.
At boot, the .so files for this sort of thing are part of the boot image and are loaded into memory, so they are available even before there's a file system. As are the userspace programs needed during startup. You can build your own boot image, with whatever programs you want. You might, for example, include the "dashboard driver" for a car, and omit terminal support. There's no assumption in QNX that there's a console, because, often, there isn't.
Some other functions, such as program loading, are in user space .so files. When you spawn a new process (in QNX, spawn is the usual function, and fork/exec are emulated for backwards compatibility), the library in your program attaches to the .so file that contains the program loader. The user process requests memory and decodes the executable image, and sets up the new process environment. If something in executable file decode does something wrong, the process faults in the usual user space way.
There's no privileged emulation layer.
At boot, the .so files for this sort of thing are part of the boot image and are loaded into memory, so they are available even before there's a file system. As are the userspace programs needed during startup. You can build your own boot image, with whatever programs you want. You might, for example, include the "dashboard driver" for a car, and omit terminal support. There's no assumption in QNX that there's a console, because, often, there isn't.
That's exactly how I did it too. The biggest hack for me was the 'open' call when using a device. That first does a lookup for a service with the name following the /dev/ bit and then it redirects the 'open' message to that process rather than to the active drive's filesystem. This allows for the appearance of a /dev/ directory without it actually existing anywhere. This makes some Unix like mechanisms hard to implement (for instance, you can't just do 'ls /dev/') but it made enough stuff work out of the box that it seemed worth the trade-off of bypassing the filesystem. It also makes the file system optional, which can come in very handy, because that means no file system is every privileged in the sense of a Unix '/' filesystem. They're all equal and you can kill each and every one of them and still have a running system.
>dive under the table every three minutes to hit the reset button
Hah, I feel that pain. Currently hacking some bare-metal Zig on PinePhone, and on top of the USB reconnect power cycle, I am transferring an SD card back n forth. I wonder if I'll be making a similar post in 30 years time.
Anyway, well done, and I look forward to browsing the code for some inspiration. Thanks for making it PD.
Hah, I feel that pain. Currently hacking some bare-metal Zig on PinePhone, and on top of the USB reconnect power cycle, I am transferring an SD card back n forth. I wonder if I'll be making a similar post in 30 years time.
Anyway, well done, and I look forward to browsing the code for some inspiration. Thanks for making it PD.
Oh good luck, I really feel your pain :) Consider using an emulator until you're really ready to go to hardware that might save you a lot of time and headache, it also makes observability a lot better. Nothing beats having access to memory from a debugger.
Thank-you for the advice. So far it has been mostly SoC stuff, and I recently got side-tracked learning some armv8 assembly, which reminds me somewhat of my 68000 days and has been good fun.
I did try QEMU for the first time last week, but started thinking I'd need some way to mock out all the hardware, and I only got as far as UART. But now that you mention it I'll take another look. There is a lot of multi-core mbox? IRQ stuff I'm about to tackle next, and I dont see why I cant develop all of that independent of the hardware.
I seem to be learning the hard way. I spent the first 12 months debugging via the only thing I could get working, an onboard RGB LED.
I did try QEMU for the first time last week, but started thinking I'd need some way to mock out all the hardware, and I only got as far as UART. But now that you mention it I'll take another look. There is a lot of multi-core mbox? IRQ stuff I'm about to tackle next, and I dont see why I cant develop all of that independent of the hardware.
I seem to be learning the hard way. I spent the first 12 months debugging via the only thing I could get working, an onboard RGB LED.
Multi-core is a level of complexity that I have no experience with, there were some i386 / i486 multi-cpu boards but I don't remember every having one of those. We did have a very early dual CPU pentium, but by that time the OS was already mothballed and we had standardized on Linux.
Interesting. Is there a picture that explains how the layers talk to each other?
Is there a VM system? How does message passing work, what kinds of protections are between tasks?
I have had many ideas for such kernels over the years, but not the patience yet to implement any of them. I wonder if Claude could help me with generating a kernel in less than a day.
I have had many ideas for such kernels over the years, but not the patience yet to implement any of them. I wonder if Claude could help me with generating a kernel in less than a day.
I should definitely make that picture. Yes, it as VM. Message passing all runs through the kernel. Task isolation is strictly memory based, whether a task acts on or responds to a message is something the tasks will have to negotiate between themselves. You could whitelist, use a token scheme or any other method for authentication that you wish to put in place, but out of the box there is none, if you know a tasks ID you can send it a message. Obviously there are ownership of resource constraints and the kernel will always ensure that the receiving task knows who the sending task is to ensure that parties are not stealing each others descriptors and such.
Pull the source and ask Claude to explain it to you...
Really interesting. Feels like a fun platform to tinker with like "circuit bending" to make surprising behaviors happen across the entire OS
I wonder what the ideal demo of this that makes its approach visually legible would be
I wonder what the ideal demo of this that makes its approach visually legible would be
That's a good question and one that I've been wondering about myself. For me it is the very clean abstraction layers. There is zero side-to-side communication it is all broader foundations to narrower verticals plugged in to those. For instance, the networking stack has a process for every layer. That keeps that whole thing manageable and it means that problems can never escape the scope of the process they are in, unlike a 'large' kernel where a botched device driver can bring down the whole system. You can pretty much tear this down to the task scheduler and the loader and rebuild it up without taking the system down. And I had a good idea on how you could replace the scheduler with another one while the system is running.
I'm really curious what - if anything - people will do with it, I think that the fact that an OS like this one powered the BlackBerry (which people loved) should work in its favor, other places where you find this kind of OS is in industrial control and in vehicles. Usually there is some kind of hardware component involved but that's entirely optional, you could use it as a general purpose OS as well.
I'm really curious what - if anything - people will do with it, I think that the fact that an OS like this one powered the BlackBerry (which people loved) should work in its favor, other places where you find this kind of OS is in industrial control and in vehicles. Usually there is some kind of hardware component involved but that's entirely optional, you could use it as a general purpose OS as well.
If you want to run this under windows then VirtualBox seems to work better than QEMU.
I would consider putting it on GitHub. Torrent is not the best way to share code today.
It's a running OS snapshot as a drive image, not a source tree, the sources are on the drive image.
As for GitHub itself: I am not going to further Microsoft's grip on the world of software development.
As for GitHub itself: I am not going to further Microsoft's grip on the world of software development.
Thank you.
For taking a stand against Microsofts "social network", "ai training data", greed and proprietary tooling.
I think sharing the source code as you do now is a perfect choice is it is what you are most comfortable with.
For taking a stand against Microsofts "social network", "ai training data", greed and proprietary tooling.
I think sharing the source code as you do now is a perfect choice is it is what you are most comfortable with.
I may end up working some more on it and releasing updated versions of the torrent. But I'm not going to be a 'maintainer', if I want unpaid work I'll volunteer at the local animal shelter instead.
Well, then go to codeberg.org