Oasis: Small statically-linked Linux system(github.com)
github.com
Oasis: Small statically-linked Linux system
https://github.com/oasislinux/oasis
224 comments
FWIW I get a faster boot time than that slimmed Debian VM with a physical Celeron class physical box booting a stock kernel, even including the real BIOS and hardware initialization time. I don't think systemd really has much to do with why that VM boots slow.
I don’t think that’s true.
That machine is a threadripper 3970x with 256G 3.2GHz DDR4, with a 4xNVMe RAID0; it’s a disturbingly fast glass-cannon.
I mean, it’s not an accurate test for sure; but I seriously doubt your VMs can boot faster stock and with the bios, on a slower machine.
That machine is a threadripper 3970x with 256G 3.2GHz DDR4, with a 4xNVMe RAID0; it’s a disturbingly fast glass-cannon.
I mean, it’s not an accurate test for sure; but I seriously doubt your VMs can boot faster stock and with the bios, on a slower machine.
Boot speeds on the timescales in the video end up having little to do with horsepower once you get to having "some sort of SSD to load the boot image from" and "some sort of CPU that can at least decompress that in a reasonable time". It's more the "Wait here for 1 second for this hardware to probe", "Wait here for 5 seconds to see if the user wants to go into setup", "Wait here for network to timeout" type configuration items. Having a ton of high end hardware just makes this worse, more cores to initialize, more memory to initialize, more devices to probe.
I picked the low end fanless box as example because of this - it boots significantly faster than my high end box after all. It still loses out to how little a VM has to do while booting but not by much. It defaults to assuming a very short "wait on user" time for BIOS/UEFI, it has a mostly static and very simple hardware topology so doesn't need to dynamically load much during BIOS/UEFI, and systemd isn't told to do all those boot time loads/delays the Debian default config seems to have. For whatever you have "slimmed down" on it you're still trying to load nvidia GPUs, run cloud-init scripts each boot, and whatever networked service you mentioned thinks it needs to wait for network before letting the box finish booting is. Those kinds of things are what make the Debian VM boot slow, not systemd or necessarily the size of the kernel until the more noticeable things are fixed. Systemd is just what Debian uses to do those boot time things and if you did the same things with OpenRC or SysVinit or your own init system without changing what was being done it'd be a pretty slow boot too.
As for VPS if "not systemd" and small/fast/static are high or your list I'd highly recommend Alpine Linux over Oasis. It's known as the gold standard for containers but it makes a nice lightweight VM box too. I don't have anything against systemd but I did run Alpine as my standard VM image for a number of years and was very happy with it on the server side plus you get to keep convenient things like a package manager if you so desire. Their "Virtual" pre-built image will have the kernel image and firmware files slimmed down to the point you might not even care about trying to roll your own changes into it.
I picked the low end fanless box as example because of this - it boots significantly faster than my high end box after all. It still loses out to how little a VM has to do while booting but not by much. It defaults to assuming a very short "wait on user" time for BIOS/UEFI, it has a mostly static and very simple hardware topology so doesn't need to dynamically load much during BIOS/UEFI, and systemd isn't told to do all those boot time loads/delays the Debian default config seems to have. For whatever you have "slimmed down" on it you're still trying to load nvidia GPUs, run cloud-init scripts each boot, and whatever networked service you mentioned thinks it needs to wait for network before letting the box finish booting is. Those kinds of things are what make the Debian VM boot slow, not systemd or necessarily the size of the kernel until the more noticeable things are fixed. Systemd is just what Debian uses to do those boot time things and if you did the same things with OpenRC or SysVinit or your own init system without changing what was being done it'd be a pretty slow boot too.
As for VPS if "not systemd" and small/fast/static are high or your list I'd highly recommend Alpine Linux over Oasis. It's known as the gold standard for containers but it makes a nice lightweight VM box too. I don't have anything against systemd but I did run Alpine as my standard VM image for a number of years and was very happy with it on the server side plus you get to keep convenient things like a package manager if you so desire. Their "Virtual" pre-built image will have the kernel image and firmware files slimmed down to the point you might not even care about trying to roll your own changes into it.
Machines with more cores and ram actually take longer to boot. I believe some of that was due to the dram training that must occur on startup, but the is must also discover all of the cores and build up the internal state to manage them. I believe some xeons take on the order of minutes to boot due to some of these considerations.
Even if that were true (which it’s not) then those points would be nullified by the fact that QEMU is exposing very little hardware (cores) and devices to the guest.
"Enterprise" class servers do a lot more in POST than consumer/desktop machines. On an older server, this can take minutes. Once the OS actually starts to boot, I have not noticed an appreciable difference.
The one on the left gets caught on raising network interfaces. I don't think your beefy machine will speed that operation up much. It wouldn't be that surprising if somebody can beat that boot time by tweaking that step (depends on the configuration -- in the extreme you don't even need a network interface or can delay it until the desktop environment is started and the user starts thinking about web browsers).
> The one on the left gets caught on raising network interfaces
I mentioned in the parent that Oasis had already booted before the delay even begins.
I also mentioned that it's not really fair to compare Debian being greatly optimised: I was measuring the practical loading time of a stock VM (that was optimised for being a VM in Debians case and which bypassed the BIOS).
The images I used were the cloud optimised ones that have a slimmed down/optimised kernel for QEMU: https://cdimage.debian.org/images/cloud/
I mentioned in the parent that Oasis had already booted before the delay even begins.
I also mentioned that it's not really fair to compare Debian being greatly optimised: I was measuring the practical loading time of a stock VM (that was optimised for being a VM in Debians case and which bypassed the BIOS).
The images I used were the cloud optimised ones that have a slimmed down/optimised kernel for QEMU: https://cdimage.debian.org/images/cloud/
But still:
> I don’t think that’s true.
There's no reason not to believe zamadatix, right? It is probably the case that they are making an irrelevant comparison, but not an untrue one.
> I don’t think that’s true.
There's no reason not to believe zamadatix, right? It is probably the case that they are making an irrelevant comparison, but not an untrue one.
I think there's sufficient reason to not believe him without evidence.
Oasis took under a second to get to a login prompt, Debian took about 1.5-2s.
UEFI BIOS takes longer than that on every system I've ever used, even little fanless ones(they weren't ever faster actually). So this claim alone makes me suspicious; but also booting faster than Oasis, on worse hardware, with more steps than my system with the slower of the two OS's I ran.
I mean, there's a point where you suspend disbelief without evidence.
Oasis took under a second to get to a login prompt, Debian took about 1.5-2s.
UEFI BIOS takes longer than that on every system I've ever used, even little fanless ones(they weren't ever faster actually). So this claim alone makes me suspicious; but also booting faster than Oasis, on worse hardware, with more steps than my system with the slower of the two OS's I ran.
I mean, there's a point where you suspend disbelief without evidence.
[deleted]
Am I misreading something? They said their system booted faster than the slow debian boot that you've shown, I think. That took (including the network time) like 8s to boot. It isn't surprising that they have a system that boots in 8s. As you note in the comment, it hangs on the network, but that's part of the boot.
It is an irrelevant comparison, but there's no reason to call them a liar over it, because it is not that surprising that a system can boot in 8s.
It is an irrelevant comparison, but there's no reason to call them a liar over it, because it is not that surprising that a system can boot in 8s.
Debian is why debian boots slow.
Oof
I'm fine with the downvotes, I didn't provide explanation. Even a casual inspection will provide basic insights such as seeing ifupdown-pre.service taking seconds on many systems.
I don't know man.
I see a display manager (lxdm under Xorg) password prompt 3 seconds after the GRUB menu.
And that includes quite a lot of stuff:
systemd-analyze blame | wc -l
72
I see a display manager (lxdm under Xorg) password prompt 3 seconds after the GRUB menu.
And that includes quite a lot of stuff:
systemd-analyze blame | wc -l
72
Normally for me on debian, just compiling the kernel to remove most modules and get rid of initrd gave great results. But the kernel config is not the same on all the machines of course.
If you want super fast boot, go with Arch Linux.
Why am I being downvoted? Is it because I didn't show proof? Here's my video showing Arch Linux (systemd) booting to the login prompt in 3 seconds, including network: https://www.dropbox.com/s/hgeopuou9rzq6ru/linux_vm_on_macos....
Perhaps the anti-systemd brigade didn't like your message ;)
Death to shared libraries. The headaches they cause are just not worth the benefit. I have been dreaming of someone cresting a system devoid of them.
> Death to shared libraries. The headaches they cause are just not worth the benefit.
Completely disagree. Even though one size does not fit all, anyone who makes sweeping statements about static libraries is just stating to the world how they are completely oblivious regarding basic software maintenance problems such as tracking which software package is updated, specially those who are not kept up to date on a daily basis.
Completely disagree. Even though one size does not fit all, anyone who makes sweeping statements about static libraries is just stating to the world how they are completely oblivious regarding basic software maintenance problems such as tracking which software package is updated, specially those who are not kept up to date on a daily basis.
Dynamic linking by default has been vocally criticized by everyone from Carmack to Bell Labs/Plan9/golang people. Many if not most of the big scale companies have historically avoided it like the plague. It has horrific costs in performance, security, reproducibility, and ultimately comprehensibility of any system.
People pay out the ass in droves via Docker (huge-ass images, clunky composability, cgroup/namespace complexity) and Nix (horror-movie learning curve) and I'm sure other things to get around it / emulate static linking. `sudo apt-whatever upgrade` and pray is pretty much DOA in even low-complexity settings. People are voting with their feet.
You can disagree, and if you're not affiliated with Debian or GNU I might even believe that you come by that opinion honestly, but this gigantic bloc of people who hate it are not all "oblivious".
People pay out the ass in droves via Docker (huge-ass images, clunky composability, cgroup/namespace complexity) and Nix (horror-movie learning curve) and I'm sure other things to get around it / emulate static linking. `sudo apt-whatever upgrade` and pray is pretty much DOA in even low-complexity settings. People are voting with their feet.
You can disagree, and if you're not affiliated with Debian or GNU I might even believe that you come by that opinion honestly, but this gigantic bloc of people who hate it are not all "oblivious".
> Docker
It's strange when an app is packaged with its own private copy of all of the "shared" libraries that it uses.
Many apps that could be a single static binary seem to be packaged as docker images largely for dependency inclusion and some cgroup/namespace isolation.
It's strange when an app is packaged with its own private copy of all of the "shared" libraries that it uses.
Many apps that could be a single static binary seem to be packaged as docker images largely for dependency inclusion and some cgroup/namespace isolation.
Bell Labs introduced shared objects into UNIX, maybe they should have learned better from other platforms.
They realized that they had goofed like 25-30 years ago and have been battling their own monster ever since: http://harmful.cat-v.org/software/dynamic-linking/versioned-....
I take the authors of a disaster more seriously when they say it's a disaster, not less.
I take the authors of a disaster more seriously when they say it's a disaster, not less.
Very few use cases that involve deliberately avoiding static linking these days imo.
I'd say this is a solved problem. It's easy to get all build dependencies of a package and for every library that gets updated you build the world.
It's a more computing power and bandwidth that is required, but the maintenance burden is mostly solved by now by modern package managers or tooling on top of not so modern ones.
It's a more computing power and bandwidth that is required, but the maintenance burden is mostly solved by now by modern package managers or tooling on top of not so modern ones.
I like the old school days of just building everything from source. Similarly the Linux Kernel was statically built -- before modules. I knew the whole system from top to bottom.
If there was a standard for specifying which static library a binary has linked, this would be a non-issue.
> If there was a standard for specifying which static library a binary has linked, this would be a non-issue.
You just gave a non-sequitur answer that rings as rational and true as stating that if the Pope would fly he would be an Airbus A330.
Limiting which static library you can link to means absolutely nothing to the problems created by static linking, and how shared libraries solve all of them without any meaningful tradeoff at all.
You just gave a non-sequitur answer that rings as rational and true as stating that if the Pope would fly he would be an Airbus A330.
Limiting which static library you can link to means absolutely nothing to the problems created by static linking, and how shared libraries solve all of them without any meaningful tradeoff at all.
[deleted]
In a sense there kinda is.
Pre-linking is a way to statically specify the lookup for a given shared library in an ELF executable. It's still kinda dynamic linking because the file is loaded at runtime but there's no library lookup, it's just "load this file at this path into address space while we load the rest of the main executable". Before PIC there would have been a bit more of a distinction but nowadays static and dynamic libraries are really only distinguished by where the symbols are (unless you want to be able to dynamically load and unload after startup).
Pre-linking is a way to statically specify the lookup for a given shared library in an ELF executable. It's still kinda dynamic linking because the file is loaded at runtime but there's no library lookup, it's just "load this file at this path into address space while we load the rest of the main executable". Before PIC there would have been a bit more of a distinction but nowadays static and dynamic libraries are really only distinguished by where the symbols are (unless you want to be able to dynamically load and unload after startup).
Dynamic linking is just poor man's IPC.
When it comes to mandatory dynamic linking, it's always a Beautiful Day for an auto-da-fé: death is too good for mandatory `glibc` and `libstdc++`.
As for a system devoid of it, I have grand plans, but for now: people do it with Docker every day! That's what Docker is. It's not broadly technically feasible right now to get a low-friction static link on a mainstream distro with mainstream libraries, but life, erm, technology Finds a Way. For now that's Docker, but the modest </s> enthusiasm around it shows that people want a way out pretty fucking bad and there will be better options soon.
As for a system devoid of it, I have grand plans, but for now: people do it with Docker every day! That's what Docker is. It's not broadly technically feasible right now to get a low-friction static link on a mainstream distro with mainstream libraries, but life, erm, technology Finds a Way. For now that's Docker, but the modest </s> enthusiasm around it shows that people want a way out pretty fucking bad and there will be better options soon.
> As for a system devoid of it, I have grand plans, but for now: people do it with Docker every day! That's what Docker is.
No, Docker really isn't what you think it is.
Docker isolates processes, and provides a convenient platform independent way to deliver stand alone applications, complete with software defined networking and file system abstraction, at the expense of shipping the whole tech stack, including everything plus the kitchen sink like full blown web servers.
Thus a .DEB package which otherwise would be 10MB ends up being a Docker image often weighing over 100MB.
And pray tell what's Docker's solution to software updates? Well, it's periodic builds that force downstream dependencies and users to re-download payloads of well over 100MB a pop.
And you advocate this as an alternative to shared libraries?
Who in their ri
No, Docker really isn't what you think it is.
Docker isolates processes, and provides a convenient platform independent way to deliver stand alone applications, complete with software defined networking and file system abstraction, at the expense of shipping the whole tech stack, including everything plus the kitchen sink like full blown web servers.
Thus a .DEB package which otherwise would be 10MB ends up being a Docker image often weighing over 100MB.
And pray tell what's Docker's solution to software updates? Well, it's periodic builds that force downstream dependencies and users to re-download payloads of well over 100MB a pop.
And you advocate this as an alternative to shared libraries?
Who in their ri
If running a webserver was as easy as
People go to ridiculous lengths just to avoid the shared library mess, and it only makes the argument against shared libraries stronger.
./nginx.bin
People go to ridiculous lengths just to avoid the shared library mess, and it only makes the argument against shared libraries stronger.
I advocate `musl`/etc. and LLVM as an alternative to `fuck.this.so.0.0.4 -> ...`. I switched from Docker to Nix some time ago (because of many of the pain points you mention).
But in a world where native applications worked on any kernel >= the one it was built on, you'd rapidly find out that bin-packing the webserver and webcrawler into a finite list of SKUs (which is where all this container mania originated) via namespaces and cgroups are really not problems that most Docker users have, and there are much simpler ways to get two versions of `node.js` looking at two different sets of libraries.
Because you can't reliably know what devilry just stomped all over your `/usr/lib/x86_64/...` or whatever, you kind of have to eat that 100MB image if you want software that worked on Tuesday to still work on Friday.
It's a sad state of affairs, but This Too Will Pass. Getting the word out that the reason Docker saves your ass on reproducibility is emulation of static linking behind the FSF's back will accelerate that process.
But in a world where native applications worked on any kernel >= the one it was built on, you'd rapidly find out that bin-packing the webserver and webcrawler into a finite list of SKUs (which is where all this container mania originated) via namespaces and cgroups are really not problems that most Docker users have, and there are much simpler ways to get two versions of `node.js` looking at two different sets of libraries.
Because you can't reliably know what devilry just stomped all over your `/usr/lib/x86_64/...` or whatever, you kind of have to eat that 100MB image if you want software that worked on Tuesday to still work on Friday.
It's a sad state of affairs, but This Too Will Pass. Getting the word out that the reason Docker saves your ass on reproducibility is emulation of static linking behind the FSF's back will accelerate that process.
> the reason Docker saves your ass on reproducibility is emulation of static linking behind the FSF's back
Can you expand on this? It’s an interesting perspective I haven’t heard before.
I get the gist but even without a Dockerfile, a container image is much less opaque and much more deterministic than a binary with a statically linked library where the compiler might inline entire functions and perform optimizations across module boundaries with LTO. It’s not as simple as setting LD_LIBRARY_PATH but it’s not like trying to reverse the information lost during compilation.
Can you expand on this? It’s an interesting perspective I haven’t heard before.
I get the gist but even without a Dockerfile, a container image is much less opaque and much more deterministic than a binary with a statically linked library where the compiler might inline entire functions and perform optimizations across module boundaries with LTO. It’s not as simple as setting LD_LIBRARY_PATH but it’s not like trying to reverse the information lost during compilation.
> Can you expand on this?
Bundling dependecies is essentially static linking that doesn't violate the GPL. Static linking means all dependencies are compiled into the binary. Modern technology like docker, appimages, snaps, etc. all emulate that by creating a static image with all the dependencies and then dynamically linking to those specific libraries. So now we have a secondary linker and loader on top of the operating system's.
Bundling dependecies is essentially static linking that doesn't violate the GPL. Static linking means all dependencies are compiled into the binary. Modern technology like docker, appimages, snaps, etc. all emulate that by creating a static image with all the dependencies and then dynamically linking to those specific libraries. So now we have a secondary linker and loader on top of the operating system's.
And it' such a shame because GNU/FSF was like the friggin Rebel Alliance fighting the Empire for the longest time: we'd never be where we are today without that stuff. Those folks were my heroes growing up.
But power corrupts, and once you've got everyone linking to your `libc`, the temptation to get some combination of lazy around standards compliance at best, and seduced by what amounts to proprietary vendor API lock-in to push your politics in at least some cases seemingly becomes overwhelming. Add in LLVM kicking your ass from here to FreeBSD with Stallman freaking the fuck out about `gcc`'s plunging relevance and there's understandable but still not acceptable instinct to go further into incompatible-by-design. People on here routinely lose their shit because NVIDIA or whoever makes it a nightmare to interoperate, and I don't like it either, but vendor lock-in is vendor lock-in.
`emacs` talks to the LSP servers via JSON for Christ's sake, which only makes it about 100x laggier, but you can see why they did it, because just like Docker even RMS can't convince anyone that What We Went With is "linking". God bless the VSCode people for accepting the performance penalty so that the rest of us can come to the party, they could have muscled LSP out of the way IMO and didn't. So `rust-analyzer` gets to shovel megabytes of JSON around and Docker gets to (figuratively) tarball a bunch of `.so` files so that `ldd` can "dynamically" link them.
The thing is, this way lies irrelevance. Even the GNU userland is pretty optional these days. Is Alpine, as I've taken to calling it, "GNU plus Linux"? Is NixOS? What are we putting up with this shit for? Bash?
But power corrupts, and once you've got everyone linking to your `libc`, the temptation to get some combination of lazy around standards compliance at best, and seduced by what amounts to proprietary vendor API lock-in to push your politics in at least some cases seemingly becomes overwhelming. Add in LLVM kicking your ass from here to FreeBSD with Stallman freaking the fuck out about `gcc`'s plunging relevance and there's understandable but still not acceptable instinct to go further into incompatible-by-design. People on here routinely lose their shit because NVIDIA or whoever makes it a nightmare to interoperate, and I don't like it either, but vendor lock-in is vendor lock-in.
`emacs` talks to the LSP servers via JSON for Christ's sake, which only makes it about 100x laggier, but you can see why they did it, because just like Docker even RMS can't convince anyone that What We Went With is "linking". God bless the VSCode people for accepting the performance penalty so that the rest of us can come to the party, they could have muscled LSP out of the way IMO and didn't. So `rust-analyzer` gets to shovel megabytes of JSON around and Docker gets to (figuratively) tarball a bunch of `.so` files so that `ldd` can "dynamically" link them.
The thing is, this way lies irrelevance. Even the GNU userland is pretty optional these days. Is Alpine, as I've taken to calling it, "GNU plus Linux"? Is NixOS? What are we putting up with this shit for? Bash?
The history of free software is the history of the GPL and the power struggles around it. Free software developers neglecting binary interfaces in favor of source code, refusing to support plugins because it enables proprietary applications, new compilers arising to challenge GCC as a result, hardware makers using cryptography to prevent users from running their own code...
And now we have this "hybrid linking" thing: static image containing dynamically linked libraries.
And now we have this "hybrid linking" thing: static image containing dynamically linked libraries.
There's a big subthread on this post about it: https://news.ycombinator.com/item?id=32459755.
Hermetic builds (Bazel, Nix, etc.) are in some sense orthogonal to packaging and delivery via e.g. a Docker container.
It's true that it takes great care to get bit-for-bit identical output from a C/C++ toolchain, and it's also true that very subtle things can send the compiler and linker into a tailspin of "now I inline this and now I don't have room to inline that and now other optimization passes do something different" and whoa, my program just got a lot faster or slower, yikes. I forget the title now, but there was a great Strange Loop talk about how if you do good statistics on repeated builds or even just stuff more or less into `argv` that the difference between `-O2` and `-O3` doesn't hold up (on either `gcc` or `clang`).
But whether the final executable is delivered to the target system via Docker or `scp` doesn't change any of that. If you want identical builds the Bazel and Nix people have a high-investment/high-return sales pitch they'd like you to hear :).
And it's also true (as another commenter pointed out) that it's at least a little bit reductionist of me to say "Docker gives you static linking". Docker does a zillion things, and for the people using the other stuff, more power to them if it's working well.
I would contend that most of them (especially since QEMU emulating x86_64 on your Intel Mac or vice versa isn't really a great story anymore) are not the real selling point though: you already had `virtualenv` and `nvm` and `rustup` and just countless ways to isolate/reproduce everything except the contents of your friendly neighborhood `/usr/lib/...`. If the native compiled code underpinning all of this stuff had all it's symbols linked in, and your kernel is the same or newer, you could use a tarball (and I've worked places where we did just that)!
`git` was probably the tool that really sold everyone on: I want a goddamned cryptographic hash of everything in the thing, otherwise I Don't Trust Like That. We don't accept mystery meat from our source code anymore, why we feel any differently about what actually runs is Stockholm's syndrome IMHO.
Hermetic builds (Bazel, Nix, etc.) are in some sense orthogonal to packaging and delivery via e.g. a Docker container.
It's true that it takes great care to get bit-for-bit identical output from a C/C++ toolchain, and it's also true that very subtle things can send the compiler and linker into a tailspin of "now I inline this and now I don't have room to inline that and now other optimization passes do something different" and whoa, my program just got a lot faster or slower, yikes. I forget the title now, but there was a great Strange Loop talk about how if you do good statistics on repeated builds or even just stuff more or less into `argv` that the difference between `-O2` and `-O3` doesn't hold up (on either `gcc` or `clang`).
But whether the final executable is delivered to the target system via Docker or `scp` doesn't change any of that. If you want identical builds the Bazel and Nix people have a high-investment/high-return sales pitch they'd like you to hear :).
And it's also true (as another commenter pointed out) that it's at least a little bit reductionist of me to say "Docker gives you static linking". Docker does a zillion things, and for the people using the other stuff, more power to them if it's working well.
I would contend that most of them (especially since QEMU emulating x86_64 on your Intel Mac or vice versa isn't really a great story anymore) are not the real selling point though: you already had `virtualenv` and `nvm` and `rustup` and just countless ways to isolate/reproduce everything except the contents of your friendly neighborhood `/usr/lib/...`. If the native compiled code underpinning all of this stuff had all it's symbols linked in, and your kernel is the same or newer, you could use a tarball (and I've worked places where we did just that)!
`git` was probably the tool that really sold everyone on: I want a goddamned cryptographic hash of everything in the thing, otherwise I Don't Trust Like That. We don't accept mystery meat from our source code anymore, why we feel any differently about what actually runs is Stockholm's syndrome IMHO.
> in a world where native applications worked on any kernel >= the one it was built on
Isn't that the world we live? Linux kernel has a stable binary interface. Problems almost always arise in user space.
Isn't that the world we live? Linux kernel has a stable binary interface. Problems almost always arise in user space.
Ah, I phrased that clumsily. Linux/Linus are fanatical about not breaking userspace (God bless them). Your stuff never runs on a different system because some asshat clobbered your `/usr/lib/...`.
> Your stuff never runs on a different system because some asshat clobbered your `/usr/lib/...`.
Yeah. I've had some success just dropping all that user space stuff and targeting Linux directly with system calls. I boot up Linux straight into my program. Not viable for every use case but it's amazing when the system is small.
Yeah. I've had some success just dropping all that user space stuff and targeting Linux directly with system calls. I boot up Linux straight into my program. Not viable for every use case but it's amazing when the system is small.
That's friggin rad. Are you at liberty to share a little about the domain that you're targeting with it? You've got to want the size/performance/etc. pretty bad to ditch every library that wants `libc` right?
I'm not using this stuff professionally, it's just my own home lab's virtual machines with little services implemented as freestanding C programs. Not doing anything fancy right now, much of it was just to see if I could do it.
I've seen other people commenting here on HN saying they're using the same approach in production so it's defenitely not my invention. Hopefully they'll see this thread and reply.
I published some of my work in the form of a liblinux that I use to make system calls:
https://github.com/matheusmoreira/liblinux
I'm not developing it anymore though because I found out the kernel itself has a superior nolibc library:
https://github.com/torvalds/linux/tree/master/tools/include/...
It used to be a single header but it looks like they've recently organized it into a proper project!
I wonder if it will become some kind of official kernel library at some point. I asked Greg Kroah-Hartman about this and he mentioned there was once a klibc:
https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroah...
I've seen other people commenting here on HN saying they're using the same approach in production so it's defenitely not my invention. Hopefully they'll see this thread and reply.
I published some of my work in the form of a liblinux that I use to make system calls:
https://github.com/matheusmoreira/liblinux
I'm not developing it anymore though because I found out the kernel itself has a superior nolibc library:
https://github.com/torvalds/linux/tree/master/tools/include/...
It used to be a single header but it looks like they've recently organized it into a proper project!
I wonder if it will become some kind of official kernel library at some point. I asked Greg Kroah-Hartman about this and he mentioned there was once a klibc:
https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroah...
That is really slick, I like your style. I could definitely live with like, one `.so` somewhere that never moves, I mean we've already got `vDSO` and stuff which is sort of related.
The quote about "glibc maintainers being non-helpful" slayed me. I guess at least some of the kernel people aren't exactly thrilled about the tireless efforts of the GNU project to gatekeep, take credit for, be obstructionist around, and otherwise subsume Linux into The GNU Operating System.
I hate that it's gotten to the point where I'm the "anti-GNU" guy, I never wanted that, but they need some fresh blood in the leadership or something.
The quote about "glibc maintainers being non-helpful" slayed me. I guess at least some of the kernel people aren't exactly thrilled about the tireless efforts of the GNU project to gatekeep, take credit for, be obstructionist around, and otherwise subsume Linux into The GNU Operating System.
I hate that it's gotten to the point where I'm the "anti-GNU" guy, I never wanted that, but they need some fresh blood in the leadership or something.
> I could definitely live with like, one `.so` somewhere that never moves, I mean we've already got `vDSO` and stuff which is sort of related.
Nice idea. I think a kernel libc in the form of a vDSO would unnecessarily force compilers to use the C ABI though. The Linux system call interface is really unique in how it's not only stable but language-agnostic and it would be a shame to lose that.
I think compilers are the right place for system calls. The programming language should have a system_call keyword that makes the compiler emit the syscall instruction with the parameters in the right registers. It was my intention to eventually create that language, liblinux was something of a prelude... Then I ran out of free time. :)
Then it turned out smarter people than me have gone much farther than I did:
https://news.ycombinator.com/item?id=28283632
Really cool stuff.
> The quote about "glibc maintainers being non-helpful" slayed me.
> I guess at least some of the kernel people aren't exactly thrilled about the tireless efforts of the GNU project to gatekeep, take credit for, be obstructionist around, and otherwise subsume Linux into The GNU Operating System.
I understand what you mean. I think it's really weird how often I get glibc manual pages when I look up Linux kernel documentation. I look up some system call and get glibc wrappers instead. Aren't they separate projects?
In the first AMA someone asked Greg whether glibc was blessed by the kernel:
https://old.reddit.com/r/linux/comments/2ny1lz/im_greg_kroah...
He said no, but it still seems like the official libc of Linux.
I also get systemd manuals when I look up Linux init system documentation. I expected to read about what the kernel itself expects from PID 1, information useful for making my own system. Turns out it's just a process like any other, it just needs to bring up the system by doing things like mounting procfs and sysfs.
Linux should be its own thing. It's only "GNU/Linux" because of POSIX which says an operating system has to have binaries like cp, mv, etc. Things don't have to be that way. POSIX is just paper.
> I hate that it's gotten to the point where I'm the "anti-GNU" guy, I never wanted that, but they need some fresh blood in the leadership or something.
I think it was a quote from Lennart Poettering that inspired me to start dropping this "standard" GNU stuff. I think he wrote something like "read TLPI and ignore all the POSIX stuff". So I did and I discovered a really nice system that actually turned out to be easier and more fun to program with than dealing with C libraries. No global state anywhere!
GNU is just a POSIX compliant user space. It doesn't have to be that way... Wouldn't it be cool if some awesome person made a pure Lisp user space? A pure Rust user space? It doesn't have to be the same old POSIX tools... Linux enables that. We can trash the entire user space and start fresh if we want. We can't do that on Windows, Mac, the BSDs.
Nice idea. I think a kernel libc in the form of a vDSO would unnecessarily force compilers to use the C ABI though. The Linux system call interface is really unique in how it's not only stable but language-agnostic and it would be a shame to lose that.
I think compilers are the right place for system calls. The programming language should have a system_call keyword that makes the compiler emit the syscall instruction with the parameters in the right registers. It was my intention to eventually create that language, liblinux was something of a prelude... Then I ran out of free time. :)
Then it turned out smarter people than me have gone much farther than I did:
https://news.ycombinator.com/item?id=28283632
Really cool stuff.
> The quote about "glibc maintainers being non-helpful" slayed me.
> I guess at least some of the kernel people aren't exactly thrilled about the tireless efforts of the GNU project to gatekeep, take credit for, be obstructionist around, and otherwise subsume Linux into The GNU Operating System.
I understand what you mean. I think it's really weird how often I get glibc manual pages when I look up Linux kernel documentation. I look up some system call and get glibc wrappers instead. Aren't they separate projects?
In the first AMA someone asked Greg whether glibc was blessed by the kernel:
https://old.reddit.com/r/linux/comments/2ny1lz/im_greg_kroah...
He said no, but it still seems like the official libc of Linux.
I also get systemd manuals when I look up Linux init system documentation. I expected to read about what the kernel itself expects from PID 1, information useful for making my own system. Turns out it's just a process like any other, it just needs to bring up the system by doing things like mounting procfs and sysfs.
Linux should be its own thing. It's only "GNU/Linux" because of POSIX which says an operating system has to have binaries like cp, mv, etc. Things don't have to be that way. POSIX is just paper.
> I hate that it's gotten to the point where I'm the "anti-GNU" guy, I never wanted that, but they need some fresh blood in the leadership or something.
I think it was a quote from Lennart Poettering that inspired me to start dropping this "standard" GNU stuff. I think he wrote something like "read TLPI and ignore all the POSIX stuff". So I did and I discovered a really nice system that actually turned out to be easier and more fun to program with than dealing with C libraries. No global state anywhere!
GNU is just a POSIX compliant user space. It doesn't have to be that way... Wouldn't it be cool if some awesome person made a pure Lisp user space? A pure Rust user space? It doesn't have to be the same old POSIX tools... Linux enables that. We can trash the entire user space and start fresh if we want. We can't do that on Windows, Mac, the BSDs.
So next time a library such as OpenSSL has a critical security vulnerability that needs to be patched, you'd rather every single binary that has it statically linked is updated, rather than a single shared library file?
Oh, yes, I certainly would. Shared libraries are an expensive, trouble-prone solution for that relatively uncommon problem.
Anyway that's not why we have shared libraries. They were invented so that memory managers could load the same library once, then map it into the address space for every process which used it. This kind of savings mattered back when RAM was expensive and executable code represented a meaningful share of the total memory a program might use. Of course that was a very long time ago now.
Anyway that's not why we have shared libraries. They were invented so that memory managers could load the same library once, then map it into the address space for every process which used it. This kind of savings mattered back when RAM was expensive and executable code represented a meaningful share of the total memory a program might use. Of course that was a very long time ago now.
> Oh, yes, I certainly would. Shared libraries are an expensive, trouble-prone solution for that relatively uncommon problem.
This personal assertion has no bearing in the real world. Please do shine some light on what exactly do you interpret as "expensive", and in the process explain why do you feel that basic software design principles like modularity and the ability to fix security vulnerabilities in the whole system by updating a single shared library is "trouble-prone".
I mean, the whole world never saw a real world problem, not the UNIX comunity nor Microsoft and it's Windows design nor Apple and it's macOS design not anyone ever, but somehow here you are claiming the opposite.
This personal assertion has no bearing in the real world. Please do shine some light on what exactly do you interpret as "expensive", and in the process explain why do you feel that basic software design principles like modularity and the ability to fix security vulnerabilities in the whole system by updating a single shared library is "trouble-prone".
I mean, the whole world never saw a real world problem, not the UNIX comunity nor Microsoft and it's Windows design nor Apple and it's macOS design not anyone ever, but somehow here you are claiming the opposite.
> the whole world never saw a real world problem, not the UNIX comunity nor Microsoft and it's Windows design
"DLL hell" [1] was absolutely a thing. It was solved only by everyone shipping the world in their application directories.
[1]: https://en.wikipedia.org/wiki/DLL_Hell
"DLL hell" [1] was absolutely a thing. It was solved only by everyone shipping the world in their application directories.
[1]: https://en.wikipedia.org/wiki/DLL_Hell
Not to mention dynamic linking is brutally slow. I don't think people even realize how many cycles are wasted. The notion that dynamic linking never had any real problems and still doesn't is totally false.
> basic software design principles like modularity
A single product can have a modular design, but modularity across unrelated products designed by different designers is a different concept that should have a different name.
In my opinion it’s just not something that really works. This is similar to the “ability” of some software to have multiple pluggable database engines. This always yields a product that can use only the lowest common denominator… badly.
But now imagine having to coordinate the sharing of database engines across multiple relying products! So upgrading Oracle would break A, B and fix C, but switching A and B to MySQL would require and update to MySQL that would break D, E and F.
That kind of thing happens all the time and people run away screaming rather than deal with it.
Shared libraries are the same problem.
A single product can have a modular design, but modularity across unrelated products designed by different designers is a different concept that should have a different name.
In my opinion it’s just not something that really works. This is similar to the “ability” of some software to have multiple pluggable database engines. This always yields a product that can use only the lowest common denominator… badly.
But now imagine having to coordinate the sharing of database engines across multiple relying products! So upgrading Oracle would break A, B and fix C, but switching A and B to MySQL would require and update to MySQL that would break D, E and F.
That kind of thing happens all the time and people run away screaming rather than deal with it.
Shared libraries are the same problem.
You aren't updating it for the whole system if some of the binaries are statically linked or have their own implementations. The point is that no matter what, you still have to audit all binaries.
Also: text pages are shared in the typical case. It was largely a spurious argument to begin with. If it didn't fuck over `musl` et. al. and `libstdc++` it would have been relegated to corner cases years ago.
Is your first sentence missing a negation, or can you elaborate?
I should have been more precise, but that can become a treatise by HN comment standards easily so consider there to be an asterisk next to all of this for myriad platform/toolchain corner cases.
To the extent that shared libraries economize on memory, or are claimed to, it's when a lot of different processes link the same version of the same library, e.g. your `libc` (or the original motivating example of X). The tradeoff here is that the kernel has an atomic unit of bringing data into RAM (AKA "core" for the old-timers), which is usually a 4kb page, and shared libraries are indivisible: you use `memcpy` and you get it and whatever else is in the 4kb chunk somebody else's linker put it in. Not a bad deal for `memcpy` because it's in everything, but a very bad deal for most of the symbols in e.g. `glibc`.
The counter-point is that with static linking, the linker brings in only the symbols you use. You do repeat those symbols for each `ELF`, but not for each process. This is often (usually?) a better deal.
A little poking around on my system to kind of illustrate this a bit more viscerally: https://pastebin.com/0zpVqA0r (apologies for the clumsy editing, it would be too long if I left in every symbol from e.g. `glibc`).
Admittedly, that's a headless machine, and some big clunky Gnome/Xorg thing might tell a different story, but it would have to be 100-1000x worse to be relevant in 2022, and almost certainly wouldn't offset the (heinous) costs. And while this is speculation, I think it would be negative worse, because there just aren't that many distinct `ELF`'s loaded on any typical system even with a desktop environment. Chromium or whatever is going to be like, all your RAM even on KDE.
There are couple of great CppCon talks that really go deep here (including one by Matt Godbolt!): https://www.youtube.com/watch?v=dOfucXtyEsU, https://www.youtube.com/watch?v=xVT1y0xWgww.
Don't be put off by the fact it's CppCon, all your GNU stuff is the same story.
To the extent that shared libraries economize on memory, or are claimed to, it's when a lot of different processes link the same version of the same library, e.g. your `libc` (or the original motivating example of X). The tradeoff here is that the kernel has an atomic unit of bringing data into RAM (AKA "core" for the old-timers), which is usually a 4kb page, and shared libraries are indivisible: you use `memcpy` and you get it and whatever else is in the 4kb chunk somebody else's linker put it in. Not a bad deal for `memcpy` because it's in everything, but a very bad deal for most of the symbols in e.g. `glibc`.
The counter-point is that with static linking, the linker brings in only the symbols you use. You do repeat those symbols for each `ELF`, but not for each process. This is often (usually?) a better deal.
A little poking around on my system to kind of illustrate this a bit more viscerally: https://pastebin.com/0zpVqA0r (apologies for the clumsy editing, it would be too long if I left in every symbol from e.g. `glibc`).
Admittedly, that's a headless machine, and some big clunky Gnome/Xorg thing might tell a different story, but it would have to be 100-1000x worse to be relevant in 2022, and almost certainly wouldn't offset the (heinous) costs. And while this is speculation, I think it would be negative worse, because there just aren't that many distinct `ELF`'s loaded on any typical system even with a desktop environment. Chromium or whatever is going to be like, all your RAM even on KDE.
There are couple of great CppCon talks that really go deep here (including one by Matt Godbolt!): https://www.youtube.com/watch?v=dOfucXtyEsU, https://www.youtube.com/watch?v=xVT1y0xWgww.
Don't be put off by the fact it's CppCon, all your GNU stuff is the same story.
Uncommon by all uses of shared libraries maybe, but certainly not uncommon for security-oriented libraries. A couple times per year is WAY more than often enough to be worth shared libraries, as otherwise you're talking about rebuilding much of the world multiple times per year per security-oriented library.
And nearly all libraries eventually become "security-oriented libraries", given enough time. Everything is a hacking target eventually.
And nearly all libraries eventually become "security-oriented libraries", given enough time. Everything is a hacking target eventually.
In real world this would just mean that more bugs are for longer unpatched.
In the majority of cases: Absolutely. That is basically the direction things have gone with "just docker build the image and redeploy".
The amount of frustration dynamic linking causes is just not worth it in today's day.
I think that calculus looks very different 20, 10, maybe even 5 years ago, when getting packages rebuilt and distributed from a zillion different vendors was much harder.
The amount of frustration dynamic linking causes is just not worth it in today's day.
I think that calculus looks very different 20, 10, maybe even 5 years ago, when getting packages rebuilt and distributed from a zillion different vendors was much harder.
This. If we'd just statically link executables, we wouldn't need Docker. Docker is a solution for a problem that only exists because of dynamic linking, and the tool's popularity should be a wakeup call.
This is painfully reductionist.
The mount namespace is only one of multiple namespaces (eg. pid, network, user, etc.) that containers utilize. Security doesn't stop at linking and shipping binaries.
That's not true.
Docker/containers do far more than just bundle shared libraries. We use a container for almost all of our CI builds. Our agents are container images with the toolchains, etc installed. It also means it's trivially easy to reproduce the build environment for those occasional wtf moments where someone has an old version of <insert tool here>.
Docker/containers do far more than just bundle shared libraries. We use a container for almost all of our CI builds. Our agents are container images with the toolchains, etc installed. It also means it's trivially easy to reproduce the build environment for those occasional wtf moments where someone has an old version of <insert tool here>.
But if the tools were all statically linked, couldn't you ship a zip with the binaries needed for building?
How do you update those tools and make sure you've deleted all the old versions? What do you do to avoid the problem of a developer SSH'ing and apt-get installing a build dependency?
Tools aren't the only part of the environment though. Setting the JAVA_HOME environment variable for a build can dramatically change the output, for example. Path ordering might affect what version of a tool is launched.
They also mean that our developers don't need to run centos 7, they can develop on windows or Linux, and test on the target platform locally from their own machine.
Tools aren't the only part of the environment though. Setting the JAVA_HOME environment variable for a build can dramatically change the output, for example. Path ordering might affect what version of a tool is launched.
They also mean that our developers don't need to run centos 7, they can develop on windows or Linux, and test on the target platform locally from their own machine.
Setting JAVA_HOME matters, again, because the Java runtime environment is essentially dynamically linked to the Java program you’re running.
The nice thing about fully statically linked executables can just be copied between machines and ran. If Java worked this way, the compiler would produce a (large) binary file with no runtime dependancies, and you could just copy that between computers.
Rust and Go don’t give you a choice - they only have this mode of compilation.
Docker forces all software to work this way by hiding the host OS’s filesystem from the program. It also provides an easy to use distribution system for downloading executables. And it ships with a Linux VM on windows and macos.
But none of that stuff is magic. You could just run a VM on macos or windows directly, statically link your Linux binaries and ship them with scp or curl.
The killer feature of docker is that it makes anything into a statically linked executable. You can just do that with most compilers directly.
The nice thing about fully statically linked executables can just be copied between machines and ran. If Java worked this way, the compiler would produce a (large) binary file with no runtime dependancies, and you could just copy that between computers.
Rust and Go don’t give you a choice - they only have this mode of compilation.
Docker forces all software to work this way by hiding the host OS’s filesystem from the program. It also provides an easy to use distribution system for downloading executables. And it ships with a Linux VM on windows and macos.
But none of that stuff is magic. You could just run a VM on macos or windows directly, statically link your Linux binaries and ship them with scp or curl.
The killer feature of docker is that it makes anything into a statically linked executable. You can just do that with most compilers directly.
java home is an example. CC is another one for most C build systems, goroot.
> But none of that stuff is magic.
I never said it was, I simply said that docker does more than statically link binaries.
>You could just run a VM on macos or windows directly, statically link your Linux binaries and ship them with scp or curl.
https://news.ycombinator.com/item?id=9224
> The killer feature of docker is that it makes anything into a statically linked executable.
The killer feature of docker is it gives you an immutable*, versionable environment that handles more than just scp'ing binaries.
> But none of that stuff is magic.
I never said it was, I simply said that docker does more than statically link binaries.
>You could just run a VM on macos or windows directly, statically link your Linux binaries and ship them with scp or curl.
https://news.ycombinator.com/item?id=9224
> The killer feature of docker is that it makes anything into a statically linked executable.
The killer feature of docker is it gives you an immutable*, versionable environment that handles more than just scp'ing binaries.
> CC is another one for most C build systems, goroot.
The benefit docker provides here is giving users a simple, reproducible, downloadable environment with no external dependencies.
I'm not saying Docker is useless in the current build environment. On the contrary - Having a convenient way to sandbox, ship and execute arbitrary execution environments and statically linked executables is really useful.
I think this is such a great idea we should make debian work closer to this model. Nix is a fantastic step in this direction and it'd be great if more distributions followed suit.
Debian and friends fill your hard drive with of weird, non-reproducible versions of various dynamically linked libraries. The fact I can't take a binary on my x86_64 linux machine and run it on your x86_64 linux machine is an embarrassment. Moving toward static linking and fully reproducible builds would be a tremendous help here.
Docker feels like a kludgy hack. I'd love a world where docker isn't necessary because its best ideas have been lowered into the OS itself.
The benefit docker provides here is giving users a simple, reproducible, downloadable environment with no external dependencies.
I'm not saying Docker is useless in the current build environment. On the contrary - Having a convenient way to sandbox, ship and execute arbitrary execution environments and statically linked executables is really useful.
I think this is such a great idea we should make debian work closer to this model. Nix is a fantastic step in this direction and it'd be great if more distributions followed suit.
Debian and friends fill your hard drive with of weird, non-reproducible versions of various dynamically linked libraries. The fact I can't take a binary on my x86_64 linux machine and run it on your x86_64 linux machine is an embarrassment. Moving toward static linking and fully reproducible builds would be a tremendous help here.
Docker feels like a kludgy hack. I'd love a world where docker isn't necessary because its best ideas have been lowered into the OS itself.
Indeed, you could. Linus takes backward compatibility more than a little seriously.
Where does static linking come into play? Why can't you ship a zip with binaries and libraries?
It's plenty common for binary distributions of software to come as tgz files. That's usually including a bunch of .so files.
It's plenty common for binary distributions of software to come as tgz files. That's usually including a bunch of .so files.
> It's plenty common for binary distributions of software to come as tgz files. That's usually including a bunch of .so files.
Yes, and compatibility across distros / releases is very hit-and-miss, because they usually ship certain libraries and rely on the OS for others.
I’m not sure whether it’s possible to ship every library without running into ODR issues, but at that point, you’d loose all of the advantages of dynamic linking anyway.
Yes, and compatibility across distros / releases is very hit-and-miss, because they usually ship certain libraries and rely on the OS for others.
I’m not sure whether it’s possible to ship every library without running into ODR issues, but at that point, you’d loose all of the advantages of dynamic linking anyway.
No idea, but it has been common since forever, and of course you miss the advantages of dynamic linking (the ability to update libraries separately) which is the whole point.
Docker provides much more isolation than a dynamic libraries.
Static linking is not a new thing. It was there since a beginning. Dynamic linking is a new thing compared to it.
Static linking is not a new thing. It was there since a beginning. Dynamic linking is a new thing compared to it.
> Docker provides much more isolation
The very opposite. You can do all the sandboxing you want without needing docker and you will avoid docker's big attack surface.
The very opposite. You can do all the sandboxing you want without needing docker and you will avoid docker's big attack surface.
Not without machine code analysis. That's a pretty big attack surface itself if you're relying on it for safety properties.
I don’t disagree, but I think there’s a productive middle ground here.
Docker has cleared become popular because it drastically eases application deployment and cleanup.
Static linking also solves a huge number of these problems, and in a generally better way.
Docker has overhead, an associated attack surface, bulky artifices, etc but it yields very nice benefits for workload isolation (env configs, storage mounts, etc) as well as cross OS/arch builds.
Is there a solution that combines the best of these options? A super-slim docker for statically linked apps that combines the ease of deployment with a much smaller footprint?
Docker has cleared become popular because it drastically eases application deployment and cleanup.
Static linking also solves a huge number of these problems, and in a generally better way.
Docker has overhead, an associated attack surface, bulky artifices, etc but it yields very nice benefits for workload isolation (env configs, storage mounts, etc) as well as cross OS/arch builds.
Is there a solution that combines the best of these options? A super-slim docker for statically linked apps that combines the ease of deployment with a much smaller footprint?
> A super-slim docker for statically linked apps that combines the ease of deployment with a much smaller footprint?
I feel like this could be a simple shell script, at least for 99% of real-world use cases.
I feel like this could be a simple shell script, at least for 99% of real-world use cases.
Not really, you'd still have a problem of building with possibly conflicting system dependencies, packaging (a dev team that can build its own .debs or .rpms is a rarity), ease of deployment / distribution, and running multiple instances. Developers don't want to deal with that crap.
That is so wrong. I use docker to build things which I cannot build on my host system because the host and build have incompatibilities. Static locking would not solve this.
[deleted]
No, both docker, static linking and flatpak share the same issue with being unable to provide reliable and tested security updates.
Once you bundle multiple libraries into blobs you always have the same security nightmare:
https://www.researchgate.net/publication/315468931_A_Study_o...
(and on top of that docker adds a big attack surface)
Once you bundle multiple libraries into blobs you always have the same security nightmare:
https://www.researchgate.net/publication/315468931_A_Study_o...
(and on top of that docker adds a big attack surface)
This argument comes up every time this discussion comes up (on flatpaks etc). And for almost all systems I’d say “yes, absolutely”.
One headache over another.
What’s to guarantee applications haven’t statically linked a library (in whatever version) rather than linked a “system” version of a library?
With the dynamic one you have both problems…
One headache over another.
What’s to guarantee applications haven’t statically linked a library (in whatever version) rather than linked a “system” version of a library?
With the dynamic one you have both problems…
Flatpaks won't be automatically recompiled though. It might take years before they bump whatever vulnerable thing they are using.
> What’s to guarantee applications haven’t statically linked a library (in whatever version) rather than linked a “system” version of a library?
It's easy to see which commands are given to the compiler and if there is a copied library if you have sources.
> What’s to guarantee applications haven’t statically linked a library (in whatever version) rather than linked a “system” version of a library?
It's easy to see which commands are given to the compiler and if there is a copied library if you have sources.
> Flatpaks won't be automatically recompiled though. It might take years before they bump whatever vulnerable thing they are using.
If we made all builds reproducible (like oasis) we could setup an automated trusted system that does builds verifiably automatically (like what Nix has).
If we made all builds reproducible (like oasis) we could setup an automated trusted system that does builds verifiably automatically (like what Nix has).
There are already trusted sources of software packages that provide security updates: traditional Linux distributions.
Doing automated, local patching and rebuilding is far from enough. You need experts to correctly backport patches and test them thoroughly.
Doing automated, local patching and rebuilding is far from enough. You need experts to correctly backport patches and test them thoroughly.
No, no… We need to move very fast, and break all the things. Tradition and conservatism is not the way. We need to compile everything and anything. Change the paradigm, break the status quo!
Oh, just like Gentoo, Slackware and LFS. ;)
Seriously, maybe we shouldn’t ignore history and learn why things are like this today.
Oh, just like Gentoo, Slackware and LFS. ;)
Seriously, maybe we shouldn’t ignore history and learn why things are like this today.
People bashing distributions seem to ignore that the alternative is the google play store.
Which reproducible builds project is striving to do for quite some time.
It’s not just a simple switch you flick and recompile.
It’s not just a simple switch you flick and recompile.
Yes but in general a system can safely be assumed to contain At least some software which was built elsewhere. Especially in more proprietary worlds like on Windows and MacOS.
> And for almost all systems I’d say “yes, absolutely”.
Do you even understand that you are advocating in favour of perpetually vulnerable systems?
Let's not touch the fact that there are vague and fantastic claims about shared libraries being this never ending sources of problems (which no one actually experiences) that when asked to elaborate the best answer is wand waving and hypotheticals and non-sequiturs.
With this fact out if the way, what is your plan to update all downstream dependencies when a vulnerability is identified?
Do you even understand that you are advocating in favour of perpetually vulnerable systems?
Let's not touch the fact that there are vague and fantastic claims about shared libraries being this never ending sources of problems (which no one actually experiences) that when asked to elaborate the best answer is wand waving and hypotheticals and non-sequiturs.
With this fact out if the way, what is your plan to update all downstream dependencies when a vulnerability is identified?
Before you decry statically linked systems as "perpetually vulnerable", I'd advise you take a look at the number of privilege escalation vulnerabilities caused by LD_PRELOAD alone.
> Before you decry statically linked systems as "perpetually vulnerable"
But statically linked systems are perpetually vulnerable by design, aren't they?
I mean, what's your plan to fix a vulnerable dependency once a patch is available?
Are you tracking each and every dependency that directly or indirectly consumes the vulnerable library? Are you hoping to have access to their source code, regardless of where they came from, and rebuild all libraries and applications?
Because with shared libraries, all it takes to patch a vulnerable library is to update just the one lib, and all downstream consumers are safe.
What's your solution for this problem? Do you have any at all?
But statically linked systems are perpetually vulnerable by design, aren't they?
I mean, what's your plan to fix a vulnerable dependency once a patch is available?
Are you tracking each and every dependency that directly or indirectly consumes the vulnerable library? Are you hoping to have access to their source code, regardless of where they came from, and rebuild all libraries and applications?
Because with shared libraries, all it takes to patch a vulnerable library is to update just the one lib, and all downstream consumers are safe.
What's your solution for this problem? Do you have any at all?
What's your plan if fixing a vulnerability requires changing the size of a struct?
> What's your plan if fixing a vulnerability requires changing the size of a struct?
Do you actually have an answer any of the points I presented you, or are you going to continue desperately trying to put up strawmen?
If any of your magical static lib promises had at bearing in the real world, you wouldn't have such a hard time trying to come up with any technical justification for them.
Do you actually have an answer any of the points I presented you, or are you going to continue desperately trying to put up strawmen?
If any of your magical static lib promises had at bearing in the real world, you wouldn't have such a hard time trying to come up with any technical justification for them.
The argument that you seem to be making is that under dynamic linking, there is never any reason to need to recompile an application, and thus the fact that static linking forces you to recompile applications makes it non-viable from a security perspective.
Unfortunately, this is very far from being true in the real world.
And even if all libraries did maintain 100% ABI compatibility forever, and even if there were never any compiler or processor bugs that needed to be mitigated by recompiling applications, dynamic linking would still add runtime complexity and thus surface area for security vulnerabilities.
> Are you tracking each and every dependency that directly or indirectly consumes the vulnerable library? Are you hoping to have access to their source code, regardless of where they came from, and rebuild all libraries and applications?
Yes? I mean, are you just throwing random untrusted binaries onto your servers, not keeping track of what their dependencies are, and hoping to God that you will never ever need to recompile them? (And if your explanation for that is "I use proprietary software from vendors who refuse to share source code", have you chosen incompetent vendors who cannot respond to security vulnerabilities in a timely fashion?)
Unfortunately, this is very far from being true in the real world.
And even if all libraries did maintain 100% ABI compatibility forever, and even if there were never any compiler or processor bugs that needed to be mitigated by recompiling applications, dynamic linking would still add runtime complexity and thus surface area for security vulnerabilities.
> Are you tracking each and every dependency that directly or indirectly consumes the vulnerable library? Are you hoping to have access to their source code, regardless of where they came from, and rebuild all libraries and applications?
Yes? I mean, are you just throwing random untrusted binaries onto your servers, not keeping track of what their dependencies are, and hoping to God that you will never ever need to recompile them? (And if your explanation for that is "I use proprietary software from vendors who refuse to share source code", have you chosen incompetent vendors who cannot respond to security vulnerabilities in a timely fashion?)
What is your plan?
Four-part plan is this:
1. For each application and library in the system, maintain a copy of the source code and the scripts necessary to rebuild it.
2. Keep track of the dependency tree of each application and library in the system.
3. If there's a problem with a dependency, update it and rebuild all dependent applications.
4. If I'm using binaries from a vendor (whether that's a Linux distribution or a proprietary software company), the vendor needs to be responsible for (1) to (3).
In practice, your package manager will generally implement almost all of (1) to (3) for you. (Newfangled systems like NPM or Cargo usually do it using lockfiles and automated tools like Dependabot.)
If you're using dynamic linking, guess what? You still have to do these four things, because even with dynamic linking, there are still problems that can only be solved by recompiling your software. Even if you think reasons like ABI incompatibility or processor bugs aren't compelling, there might be, y'know, bugs in the applications themselves that you've gotta patch.
1. For each application and library in the system, maintain a copy of the source code and the scripts necessary to rebuild it.
2. Keep track of the dependency tree of each application and library in the system.
3. If there's a problem with a dependency, update it and rebuild all dependent applications.
4. If I'm using binaries from a vendor (whether that's a Linux distribution or a proprietary software company), the vendor needs to be responsible for (1) to (3).
In practice, your package manager will generally implement almost all of (1) to (3) for you. (Newfangled systems like NPM or Cargo usually do it using lockfiles and automated tools like Dependabot.)
If you're using dynamic linking, guess what? You still have to do these four things, because even with dynamic linking, there are still problems that can only be solved by recompiling your software. Even if you think reasons like ABI incompatibility or processor bugs aren't compelling, there might be, y'know, bugs in the applications themselves that you've gotta patch.
> If I'm using binaries from a vendor (whether that's a Linux distribution or a proprietary software company), the vendor needs to be responsible for (..)
I'm not even going to bother pointing how unfeasible all your other points are. I'm just going to point this Fack: you do understand this does not work and never worked at any point in time, don't you?
Do you understand the explicit reference to perpetually vulnerable systems? Do you realize where it cames from?
You only have the power to rebuild the packages you own personally, and even so static libs offer zero ways to keep track of which version went into which build.
Once your fantastic panacea starts to rely on your idea to force third parties to follow your personal orders to make new releases under your own personal terms, you should be very aware that you will not get your wish. You'll instead just keep on using the same vulnerability-riddled release.
Do you understand the importance and value of static libs? You do not need four-point authoritarian and deeply impractical and unfeasible plans to keep your system safe. With shared libraries you just patch the one lib, and all your system is safe.
And again what tradeoff do you want to achieve for this? Nothing?
I'm not even going to bother pointing how unfeasible all your other points are. I'm just going to point this Fack: you do understand this does not work and never worked at any point in time, don't you?
Do you understand the explicit reference to perpetually vulnerable systems? Do you realize where it cames from?
You only have the power to rebuild the packages you own personally, and even so static libs offer zero ways to keep track of which version went into which build.
Once your fantastic panacea starts to rely on your idea to force third parties to follow your personal orders to make new releases under your own personal terms, you should be very aware that you will not get your wish. You'll instead just keep on using the same vulnerability-riddled release.
Do you understand the importance and value of static libs? You do not need four-point authoritarian and deeply impractical and unfeasible plans to keep your system safe. With shared libraries you just patch the one lib, and all your system is safe.
And again what tradeoff do you want to achieve for this? Nothing?
That's like claiming that a leaky sump pump sank the titanic.
Linux has never been anywhere close to resistant to privilege escalation bugs.
Linux has never been anywhere close to resistant to privilege escalation bugs.
Google “OpenSSL dependency conflict” and view the results. There is significantly more work going on when OpenSSL releases and update than “just update the single shared library file.” Applications often need to be updated as well, so you may as well just statically link the new binary.
Haven't seen a single OpenSSL issue over the past 25 years of using Linux.
libstdc++ errors? Seen them.
libstdc++ errors? Seen them.
Libstdc++ errors notwithstanding, what about heartbleed?
https://en.m.wikipedia.org/wiki/Heartbleed
https://en.m.wikipedia.org/wiki/Heartbleed
I think the parent was talking about dependency conflicts, not issues in OpenSSL.
It really depends on whether the bug is in API or not. If it is not, it is likely that upgrading only the shared library is enough.
How lovely to live in a world where api contracts are absolute and complicated internal behaviors never conflict in unforeseen ways. We’ve really gotten on top of the software complexity problem
> How lovely to live in a world where api contracts are absolute and complicated internal behaviors never conflict in unforeseen ways.
Some libraries managed to preserve API contracts throughout the years just fine.
Even in C++, a language which does not establish a standard ABI, has frameworks which have not been breaking the ABI throughout multiple releases, such as Qt.
Some libraries managed to preserve API contracts throughout the years just fine.
Even in C++, a language which does not establish a standard ABI, has frameworks which have not been breaking the ABI throughout multiple releases, such as Qt.
That's exactly the problem deb files and apt solve (via integration testing and documenting antidependences.)
> you'd rather every single binary that has it statically linked is updated, rather than a single shared library file?
Yes! A thousand times yes. I'd rather reinstall every single compromised program than deal with the ridiculous complexity of dynamic linking.
Yes! A thousand times yes. I'd rather reinstall every single compromised program than deal with the ridiculous complexity of dynamic linking.
In some cases it’s impossible to build older software, or very difficult.
Considering the complexity that shared libraries add over static linking: yes absolutely.
Is that only that good libraries should correctly use semver (i.e. be backward-compatible unless they change the major version), or is there complexity somewhere else?
My feeling is that dynamic linking is a pain because sometimes one needs, say, 1.2.5 for some part of the code and 1.3.1 for another part, and they are not compatible. Which hints towards the library author having messed up something, right?
My feeling is that dynamic linking is a pain because sometimes one needs, say, 1.2.5 for some part of the code and 1.3.1 for another part, and they are not compatible. Which hints towards the library author having messed up something, right?
We just didn't update Log4j since it broke a critical, old, closed source offline program. Vendor has long dropped support for this old program. Cue that up to the usual executive missteps that we get from up high.
Yay, we get to keep on using old Log4j now because one program holds it back.
Yay, we get to keep on using old Log4j now because one program holds it back.
If the "old log4j" is 1.2.x, you can update to reload4j (https://reload4j.qos.ch/), which is a fork of the latest log4j 1.2.x by its initial author; you don't have to update to log4j 2.x (which changed the API). And you might not even need to update, since "old log4j" (1.2.x) wasn't affected by that vulnerability in the first place (it has its own old vulnerabilities, but they're all in code which is usually not used by real world deployments).
Yes. I can think of worse problems w shared libs. Should we start a list?
If you keep around the .o / .a files this isn't so bad.
you only need to relink, not recompile. i think oasis caches artifacts so there's no overhead.
Sure. Networks are fast enough.
What about closed-source binaries where a patched binary can't be obtained from the vendor?
You could link locally, or you could opt not to use them. Or you could restrict dynamic linking to these special cases.
The same thing I do if the vulnerability is in the binary itself, and not in a library.
Surely, if you care, you have a mitigation plan for this?
Surely, if you care, you have a mitigation plan for this?
Binary patch it.
With closed-source binaries the argument for static linking is the strongest, IMHO, or at least shipping all the required shared libraries and LD_LIBRARY_PATH that in. The amount of times I've had to muck about to make a five-year old game work on a current system is somewhat bonkers, and I would never expect someone unfamiliar with shared library details (i.e. a normal non-tech person, or even a tech person not familiar with C/Linux) to figure all of that out.
Closed source programs almost universally bundle all dependencies (except system ones) irrespective of whether they are using static or dynamic linking. They have to to ensure that they actually run reliably.
Or you can link locally.
Yes
> I have been dreaming of someone cresting a system devoid of them.
Doesn’t that pretty much already exist now with immutable system builds, eg Fedora Silverblue, GUIX, NixOS? Each package pulls in its own dependencies with the specified version and config, and then no changes can be made once the system is built. Only way to update is to rebuild the system with the updated versions. Costs more disk space but that is cheap these days, and solves shared dependency hell.
Doesn’t that pretty much already exist now with immutable system builds, eg Fedora Silverblue, GUIX, NixOS? Each package pulls in its own dependencies with the specified version and config, and then no changes can be made once the system is built. Only way to update is to rebuild the system with the updated versions. Costs more disk space but that is cheap these days, and solves shared dependency hell.
You're just embracing another set of problems. Shared libraries were introduced for a reason. The are complex to handle, but that's a long standing tooling problem rather than an inherent issue of the model. I think we're actually just getting the hang of it with stuff like Nix and containers.
“Shared libraries were introduced for a reason.”
I’m pretty sure the reason is “X11 was large and badly factored, and people who should have known better thought it would save disk space.”
The problems they claim to solve are updates across applications at once, but in the enterprise how often does that actually happen? Application authors still have to test against the new version, and sysadmins still have to roll out the changes required to support the update. In practice they may as well statically link.
Let’s not forget the problems they introduce are significant. Implementations are complex. Applications become less portable. ABI versioning is a pain (versioned symbols are gross). Semantic changes in the shared lib cause chaos. Exec is slower because of the linking done at load time. Dynamic libs are bigger than static ones, AND they’re loaded into memory. More difficult to verify the correctness of a given executable.
The alleged “benefit” here really does not outweigh the cost. If we were doing static linking from the beginning, it’s difficult to say if we would have wound up with things like docker or nix in the first place.
I’m pretty sure the reason is “X11 was large and badly factored, and people who should have known better thought it would save disk space.”
The problems they claim to solve are updates across applications at once, but in the enterprise how often does that actually happen? Application authors still have to test against the new version, and sysadmins still have to roll out the changes required to support the update. In practice they may as well statically link.
Let’s not forget the problems they introduce are significant. Implementations are complex. Applications become less portable. ABI versioning is a pain (versioned symbols are gross). Semantic changes in the shared lib cause chaos. Exec is slower because of the linking done at load time. Dynamic libs are bigger than static ones, AND they’re loaded into memory. More difficult to verify the correctness of a given executable.
The alleged “benefit” here really does not outweigh the cost. If we were doing static linking from the beginning, it’s difficult to say if we would have wound up with things like docker or nix in the first place.
Surely the main reason is that the linked library may actually be different on different machines. Stuff like Open GL.
> Exec is slower because of the linking done at load time.
Oh, cry me a river. 30 years ago the performance impact was known and was very well worth the trade-off. Machines are orders of magnitude faster now and consequently the impact less significant. Next thing, you're telling me X11 is slow.
Oh, cry me a river. 30 years ago the performance impact was known and was very well worth the trade-off. Machines are orders of magnitude faster now and consequently the impact less significant. Next thing, you're telling me X11 is slow.
Plenty of support for static linking, I am seeing.
Would static linking conceivably make sandboxing, jailing, etc easier?
Compared to all the effort which goes into dynamic linking, I suspect it would be better spent on sandboxing (both mechanism and UX).
Would static linking conceivably make sandboxing, jailing, etc easier?
Compared to all the effort which goes into dynamic linking, I suspect it would be better spent on sandboxing (both mechanism and UX).
> Shared libraries were introduced for a reason.
Those reasons were floppy disks and RAM that was measured in kilobytes though.
Those reasons were floppy disks and RAM that was measured in kilobytes though.
nix and containers actually eliminate all the benefits of shared libraries while still having all their problems
Non-deterministic and unpredictable behaviour is a bigger problem than whatever benefits you think shared libraries provide. That's why so many people here prefer static linking, and that's what Nix gives you even for dynamic linking.
Yes. But the other set of problems exists anyway. Even if you drop in a new system version of a shared library, you still need to consider what other versions are used apart from the system library. Any app (other than those provided by a system
package manager of course) can have a local version different from the system version and any app can have linked it statically?
>Shared libraries were introduced for a reason.
Lack of storage and bandwidth in the 1970s
Lack of storage and bandwidth in the 1970s
False. The main reason was to avoid having to maintain and update multiple copies.
[deleted]
What are the headaches with shared libraries?
Assuming that you pin the version of the library you need, two applications using the same library /shouldn't/ cause issues.
In most *nix systems - each instance of a shared library has its own data.
Assuming that you pin the version of the library you need, two applications using the same library /shouldn't/ cause issues.
In most *nix systems - each instance of a shared library has its own data.
I am assuming you know the answer to the question if you have been developing code for awhile. What makes the problem worse is that it is transitive. I link against Colmap libs, which links against Ceres libs, which links against OpenMP, … these are ever changing code bases and I have to create executables that run on local machines, Azure cloud machines, AWS cloud machines, … chafed down shared lib mismatches becomes a nightmare.
“ Assuming that you pin the version of the library you need, two applications using the same library /shouldn't/ cause issues.”
Then why bother?
Then why bother?
I used to feel the same way. In an ideal world all packages would be automated and developers would have a way to alert packaging tools that a new version is release, so that it can be run through CI/CD and users would get new packages almost instantaneously without any breaking changes.
However, until that happens there is nothing wrong with shared libraries. They exist for good reason. They save bandwidth and ensure that you're not getting old libraries.
However, until that happens there is nothing wrong with shared libraries. They exist for good reason. They save bandwidth and ensure that you're not getting old libraries.
The save bandwidth? Do you mean memory? You want to link against the exact same code you developed and tested your program with. Ensuing you get and exact match is critical for robust code.
Yes, dynamic libraries save a descent amount of bandwidth when installing packages. Also, for the most part no I don't want the exact libraries always. If library creators have a security fix then I want the one with the fix, and there are plenty of times I've seen developers create something and barely maintain it afterwards. Package maintainers are really good at keeping packages for necessary libraries.
You could try DEC Ultrix. (Oh, did you mean a modern system?)
I dont disagree. But wondering if I am missing anything other than the usual cons such as Memory usage, code ( Storage ) size and longer compile time.
I understand the motivation for shared libs is the reduced memory footprint for large common libs. I have an alternative idea that avoids the headaches of shared lib mismatches and still can reduce the memory footprint. Have relocatable code be statically linked but stored with a hash. When loading a binary see if common relocatable code is already loaded .. if it has the same hash then it is exactly the same and can be safely shared. No version mismatch headaches.
> Have relocatable code be statically linked but stored with a hash. When loading a binary see if common relocatable code is already loaded .. if it has the same hash then it is exactly the same and can be safely shared.
It's not actually "code" which is being shared, but instead the memory pages containing that code (the sharing granularity is a single page, since sharing works by pointing the page table entries on each process to the same physical page). So either you'd have to force page alignment for each and every function (highly wasteful, since the page size is at least 4096 bytes), or you're reinventing shared libraries but using a hash instead of a version.
It's not actually "code" which is being shared, but instead the memory pages containing that code (the sharing granularity is a single page, since sharing works by pointing the page table entries on each process to the same physical page). So either you'd have to force page alignment for each and every function (highly wasteful, since the page size is at least 4096 bytes), or you're reinventing shared libraries but using a hash instead of a version.
This is not like shared libraries in that my program is guaranteed to load and run regardless of what libs are on the system. Also, I am guaranteed my program is running with the exact same library I tested it with.
That's finer grained than I imagined. I am assuming hashes for large pieces of code that are completely relocatable in memory.
I think often the binaries will have been compiled against slightly different versions. With dynamic linking, I can swap 1.0.1 for 1.0.2 or 1.0.3 without issue. The hash would require an exact match, which is likely rare.
I want an exact match. That is what I developed and tested against. A change, however small, can have unintended consequences. Changing versions is a huge issue from a reliability point of view.
Wouldn't you want to randomise some stuff to make attacks harder? (Sorry, I forgot the details.)
Which is not feasible for misters like qt WebKit or chromium.
Absolutely not. Dynamic libraries are crucial for security.
Static linking again and again has proved to be a security nightmare. It makes security updates dependent from upstream release or full rebuilds making things difficult.
Static linking again and again has proved to be a security nightmare. It makes security updates dependent from upstream release or full rebuilds making things difficult.
Proved to whom, when, and in what context? Systems that were designed for easy reproducible builds from the ground up are fairly new, so how sure are you the context hasn't changed?
But aren't you dependent from upstream anyway? Or do you recompile from source the broken dynamic library? In my experience, most sysadmins these days don't compile from source.
No to both. The Linux distribution you use is responsible for backporting patches, fixing the vulnerability, building updated packages, testing them and sending the updates.
You get targeted security updates without changes in any other functionality.
You get targeted security updates without changes in any other functionality.
> BearSSL is incredibly small and well written, but is not widely adopted.
Well, probably because the author himself discourages you from using it in anything serious. Crypto libraries need a lot of testing and reviews by experts, which BearSSL doesn't have yet. Thomas Pornin is an incredibly smart guy when it comes to crypto and BearSSL is my crypto library of choice for small (mostly embedded) projects, too. Projects like this might help to get it the attention it needs.
Well, probably because the author himself discourages you from using it in anything serious. Crypto libraries need a lot of testing and reviews by experts, which BearSSL doesn't have yet. Thomas Pornin is an incredibly smart guy when it comes to crypto and BearSSL is my crypto library of choice for small (mostly embedded) projects, too. Projects like this might help to get it the attention it needs.
I use BearSSL in a few toy projects, would love to see it get more traction and eventually some proper cryptographic audit if it moves out of alpha.
I'm curious how it handles the client side dns resolver. That's one of the sticking points for "fully static linking".
Edit: Looks like it uses musl's resolver. Which is better than I remember it when using it a long time ago. It does still lack some things like IDN/punycode support. And things like this: https://twitter.com/richfelker/status/994629795551031296?lan...
Pluggable authentication modules (PAM) may also be an issue with this setup, probably at least requires a lot of manual research and twiddling.
Edit: Looks like it uses musl's resolver. Which is better than I remember it when using it a long time ago. It does still lack some things like IDN/punycode support. And things like this: https://twitter.com/richfelker/status/994629795551031296?lan...
Pluggable authentication modules (PAM) may also be an issue with this setup, probably at least requires a lot of manual research and twiddling.
The musl resolver also lacks "no-aaaa" which glibc grew a week or two ago.
This line is a bit concerning to me: "and libtls-bearssl, an alternative implementation of libtls based on BearSSL" It sounds like a custom rolled variant of a low-adoption SSL lib that isn't itself very well tested.. It may be better.. it may be horribly flawed or backdoored in a non-obvious way. But I also get how horrible OpenSSL is..
same here, in fact the last bearssl release was 4 years ago, very inactive if still alive.
mbedtls might be better here, but it does not have tls1.3 fully ready(I don't think bearssl has 1.3 either).
wolfssl could be the best, small and robust with new features, it is dual licensed if you want to use it in commercial releases.
mbedtls might be better here, but it does not have tls1.3 fully ready(I don't think bearssl has 1.3 either).
wolfssl could be the best, small and robust with new features, it is dual licensed if you want to use it in commercial releases.
I'd love to see someone port this to Bazel and build a ruleset to generate os images for KVM/VirtualBox/flashing to bare metal. This would be great for having a generic platform for building embedded systems from source.
I'm working on a Nix/NixOS based polyglot Bazel build infrastructure (that we're using in production) and I've almost got `glibc` and `libstdc++` squeezed all the way out. They're like fucking Kudzu (which is not an accident), but I'll get 'em eventually. Once `glibc` and `libstdc++` are all the way gone it will be a hop, skip, and a jump to doing what you're describing.
I anticipate it being ready for open source as a standalone flake by the end of 2022.
I anticipate it being ready for open source as a standalone flake by the end of 2022.
I literally had that conversation in their IRC channel and the resounding consensus was that “why would you want that”
I was asking if anybody had done it yet, but since there’s people who want that I will submit some BUILD field to oasis. :)
I was asking if anybody had done it yet, but since there’s people who want that I will submit some BUILD field to oasis. :)
I'm really grateful for Michael Forney's work on this project since I found it a good resource for getting a variety of popular packages to build statically.
I ended up starting my own statically-linked Linux distro, but with some creative differences like shell scripts (vs Lua scripts) and package management. I recently "finished" it (close enough anyway) to the point that I run it as my daily driver.
Not trying to plug my project (it's not public, not yet anyway) but if anyone wants to discuss this sort of thing (distro creation and static linking in particular) and compare notes via email, I'd be happy to.
I ended up starting my own statically-linked Linux distro, but with some creative differences like shell scripts (vs Lua scripts) and package management. I recently "finished" it (close enough anyway) to the point that I run it as my daily driver.
Not trying to plug my project (it's not public, not yet anyway) but if anyone wants to discuss this sort of thing (distro creation and static linking in particular) and compare notes via email, I'd be happy to.
“I tend to think the drawbacks of dynamic linking outweigh the advantages for many (most?) applications.” – John Carmack
<rant>
I don't like to be a parrot for the Plan9 people [1] (they've been winning their own fights for a while now), but they've assembled a better set of citations than I could in the time I'm willing to devote to an HN comment.
Dynamic linking is fine, in very specific, very limited cases. Python extensions. Maybe you want to do a carve-out for like, some small hardened core of an SSL implementation or a browser because you've got actual hot-deploy in your (massive) installation/infrastructure.
As a basically mandatory default it's a chemical fire next to an elementary school. It started gaining serious traction as a horrific hack to get a not-remotely-finished X Windows to run on Sun boxes in like, +/- 1990. It wasn't long before the FSF had figured out that they could bio-weaponize it against interoperability. The `glibc` docs are laughing in their beer while they explain why static linking on "GNU" systems is impossible, and also makes you a criminal. They don't even pretend to make a cogent argument (they said the quiet part loud on the mailing lists). It's to kill other `libc` and `libc++` (fuck you small-but-vocal anti-interoperability FSF people!). Don't want LLVM to finish you off? Start interoperating better. There are a lot of us who think "Free Software" has to mean that I can use it in legal ways that you philosophically disagree with, and if it happens to be better than your Win32-style lock-in? Well that's just groovy.
Dynamic linking by (mandatory) default: Just Say No.
Myths:
- People want it: Docker. QED.
- Secure: Dynamic linking is the source of tons of security vulnerabilities. More generally, the best way to not trust your system is to not be able to audit the literal code you're running without jumping through insane, sometimes practically impossible hoops. If you want "father knows best" security, you need self-patch like Chromium and shit do now. And if you're willing to trade performance for security by the gross, flat pack or something.
- Performant: Dynamic linking forces address patch-ups, indirect calls, and defeats many optimizations. It thrashes TLB. It slows startup of even simple programs.
- Necessary: Text pages are shared. I don't care if you've got 9, 90, or 90k processes using `printf`: it's going to be in the same text page unless something else went badly, badly wrong.
</rant>
[1] http://harmful.cat-v.org/software/dynamic-linking/
<rant>
I don't like to be a parrot for the Plan9 people [1] (they've been winning their own fights for a while now), but they've assembled a better set of citations than I could in the time I'm willing to devote to an HN comment.
Dynamic linking is fine, in very specific, very limited cases. Python extensions. Maybe you want to do a carve-out for like, some small hardened core of an SSL implementation or a browser because you've got actual hot-deploy in your (massive) installation/infrastructure.
As a basically mandatory default it's a chemical fire next to an elementary school. It started gaining serious traction as a horrific hack to get a not-remotely-finished X Windows to run on Sun boxes in like, +/- 1990. It wasn't long before the FSF had figured out that they could bio-weaponize it against interoperability. The `glibc` docs are laughing in their beer while they explain why static linking on "GNU" systems is impossible, and also makes you a criminal. They don't even pretend to make a cogent argument (they said the quiet part loud on the mailing lists). It's to kill other `libc` and `libc++` (fuck you small-but-vocal anti-interoperability FSF people!). Don't want LLVM to finish you off? Start interoperating better. There are a lot of us who think "Free Software" has to mean that I can use it in legal ways that you philosophically disagree with, and if it happens to be better than your Win32-style lock-in? Well that's just groovy.
Dynamic linking by (mandatory) default: Just Say No.
Myths:
- People want it: Docker. QED.
- Secure: Dynamic linking is the source of tons of security vulnerabilities. More generally, the best way to not trust your system is to not be able to audit the literal code you're running without jumping through insane, sometimes practically impossible hoops. If you want "father knows best" security, you need self-patch like Chromium and shit do now. And if you're willing to trade performance for security by the gross, flat pack or something.
- Performant: Dynamic linking forces address patch-ups, indirect calls, and defeats many optimizations. It thrashes TLB. It slows startup of even simple programs.
- Necessary: Text pages are shared. I don't care if you've got 9, 90, or 90k processes using `printf`: it's going to be in the same text page unless something else went badly, badly wrong.
</rant>
[1] http://harmful.cat-v.org/software/dynamic-linking/
> The `glibc` docs are laughing in their beer while they explain why static linking on "GNU" systems is impossible, and also makes you a criminal.
Can you provide a reference? I skimmed through the glibc manual but couldn't find anything like this.
> It's to kill other `libc` and `libc++`
How does this follow from dynamic linking?
Can you provide a reference? I skimmed through the glibc manual but couldn't find anything like this.
> It's to kill other `libc` and `libc++`
How does this follow from dynamic linking?
Good primary sources are the `glibc` documentation, especially the FAQ [1], and the `musl` docs, especially the "differences from glibc" section [2] and the FAQ [3], which is basically a long list of ways that `glibc`/`gnulib` developers have leaned into ever-more-broken/nonstandard behavior. If you dig through the GNU mailing lists (which is where I draw the line for an HN comments), you'll find megabytes of admission that this is in an effort to make "GNU/Linux" (or as I've taken to calling it, GNU "plus" Linux, Linux being merely the kernel...), to quote Fassbender as Steve Jobs: "Closed system! Completely incompatible with anything!"
But the short story is that both NSS (DNS lookup), and `iconv` (locale handling) don't work without `glibc`-specific dynamic linking shit. Ostensibly this is because of things like merely being able to read decades-obsolete multibyte character encodings, and not able to write them, is a bridge too far to support at all, let alone as a default. They've also got a bridge they'd like to sell you (see mailing lists).
It's just 90s Microsoft vendor lock-in: "embrace, extend, extinguish" we used to call it. Stallman was a hero for getting the free software movement rolling in the 80s. But in this business, you either die a hero or live long enough to become the villain, and Bill Gates himself would be proud.
[1] https://sourceware.org/glibc/wiki/FAQ [2] https://wiki.musl-libc.org/functional-differences-from-glibc... [3] https://wiki.musl-libc.org/faq.html
But the short story is that both NSS (DNS lookup), and `iconv` (locale handling) don't work without `glibc`-specific dynamic linking shit. Ostensibly this is because of things like merely being able to read decades-obsolete multibyte character encodings, and not able to write them, is a bridge too far to support at all, let alone as a default. They've also got a bridge they'd like to sell you (see mailing lists).
It's just 90s Microsoft vendor lock-in: "embrace, extend, extinguish" we used to call it. Stallman was a hero for getting the free software movement rolling in the 80s. But in this business, you either die a hero or live long enough to become the villain, and Bill Gates himself would be proud.
[1] https://sourceware.org/glibc/wiki/FAQ [2] https://wiki.musl-libc.org/functional-differences-from-glibc... [3] https://wiki.musl-libc.org/faq.html
I understand why, technically speaking, glibc doesn't support static linking (and I also think that it's bad design). What I don't understand is how this makes it "incompatible with anything". The glibc-specific APIs, sure, they do - to the extent the apps actually use them (which, judging by the intersection of e.g. Debian and FreeBSD package repos, is not exactly common). But how does denying the ability to link statically promote this goal?
Your last point is a bit off. Text pages are shared iff they come from the same file. So your point works if the 90k processes are the same program. Else: shared libs...
Yeah I was typing too fast in ranty mode, thanks for calling it. I did a much more detailed answer that gets it (mostly) right replying to another commenter: https://news.ycombinator.com/item?id=32461766
The core values are very appealing to me.
what is the typical update routine a la 'sudo apt-get update && sudo apt-get dist-upgrade'?
what is the typical update routine a la 'sudo apt-get update && sudo apt-get dist-upgrade'?
From the readme:
No package manager.
Instead, you configure a set of specifications of what files from which packages to include on your system, and the build system writes the resulting filesystem tree into a git repository. This can then be merged into /, or pulled from another machine.
Debian has reproducable builds going for it. I would be very interested in static gentoo like source distro though.
As someone who clearly doesn't understands this well enough - can I run a desktop environment on top of it?
it has Velox which is a Wayland compositor, so you can start running static linked Wayland programs.
and of course you can add things like Firefox, LibreOffice or whatever, but you either need to spend many many cursed nights and dark days to get them to compile statically ... or just add a dynamic linker (ld) and add the dependencies... but at that point you built a second independent system on the same filesystem.
and of course you can add things like Firefox, LibreOffice or whatever, but you either need to spend many many cursed nights and dark days to get them to compile statically ... or just add a dynamic linker (ld) and add the dependencies... but at that point you built a second independent system on the same filesystem.
Is this meant more to be used on a small embedded Linux system? As a Serverless OS for fast startup?
Would dlopen be disallowed on this system?
E.g., technically speaking most (all?) audio plug-ins are shared libs...
E.g., technically speaking most (all?) audio plug-ins are shared libs...
I was ready to give this a try, but the use of BearSSL is concerning. It goes on to say that it's not a widely adopted SSL implementation, so how can it be trusted? I suppose adoption has to start somewhere, but I'll watch from a safe distance.
I used to audit tls libraries for a living and bearssl is the library I would choose today if I had to make a choice
Can you elaborate?
Interesting project but according to the github history it has had only one official release in 2017 and there has been only 34 commits so far this year. Is the project in low maintenance mode?
The discussions on static- vs dyamic linking piqued my interest. Is there any resources I should read about the history and concrete pros & cons of linking techniques?
Sounded good to me, until I found out there's no repository; you just pull everything from github. Then I stopped reading. What could go wrong?
>and is probably better compared to a BSD.
That's a very unfortunate comment in the second paragraph.
That's a very unfortunate comment in the second paragraph.
You’re misreading it. The author means that it’s more similar to a BSD than to a typical Linux system, as the previous sentence makes clear.
((better compared) (to BSD)), not (better (compared to BSD)) :)
Or:
"[...] is better compared to BSD", not "[...] is better, compared to BSD".
"[...] is better compared to BSD", not "[...] is better, compared to BSD".
Yeah, that's definitely a better way to put it.
Should probably clarify in what ways (or for what) it's supposedly better. That's quite the inflammatory statement.
I don't read that sentence to mean it's better than BSD, rather that Oasis is more like BSD than a typical Linux distribution.
"It is better compared to" is an idiom meaning "the comparison is more apt to".
"It is better compared to" is an idiom meaning "the comparison is more apt to".
[deleted]
phrasing should be changed to "similar to BSD in design".
I used this before, pretty impressive how much it can pull off
> Minimal bootstrap dependencies.
Any POSIX system with git, lua, curl, a sha256 utility, standard compression utilities, and an x86_64-linux-musl cross compiler can be used to bootstrap oasis.
So you need a custom POSIX system to bootstrap oasis.
> All software in the base system is linked statically,
this could be a good thing, however
> including the display server (velox) and web browser (netsurf).
No X server and an obscure web browser. Useless.
So you need a custom POSIX system to bootstrap oasis.
> All software in the base system is linked statically,
this could be a good thing, however
> including the display server (velox) and web browser (netsurf).
No X server and an obscure web browser. Useless.
Name me an OS that doesn't need an OS to build it.
It needs a very special OS: "git, lua, curl, a sha256 utility, standard compression utilities, and an x86_64-linux-musl cross compiler"
That's... a list of things I already have installed, the Lua executable aside. This is a pretty normal-looking dependency list for native C software. Are you perhaps not used to building C software?
Of those, only an x86_64-linux-musl cross compiler strikes me as even remotely unusual, and that's still widely available.
You mean besides just install the relevant packages by: "pacman -S git lua52 gzip coreutils gcc musl" on a standard arch install? (presumably similar for other distros)
> No X server and an obscure web browser. Useless.
Indeed, it uses a custom Wayland compositor. To me it looks like using a minimal Wayland compositor would fit the project better than using X. The same could be said about using Netsurf.
Indeed, it uses a custom Wayland compositor. To me it looks like using a minimal Wayland compositor would fit the project better than using X. The same could be said about using Netsurf.
Velox appears to be a Wayland compositor, and supports Xwayland.
um, it says if you have the libs and configured compiler, you can build it.
there's also nothing stopping you from doing static builds of whatever application you want to include in your OS.
there's also nothing stopping you from doing static builds of whatever application you want to include in your OS.
Obviously I'm writing this because Oasis absolutely trounced it.
So; Oasis is currently my go-to for small deployed VPS's.
(here's the video if you actually care: https://www.youtube.com/watch?v=X2Aw5SSqYFo you'll note that the systemd one was delayed by the network on start- some people are going to argue that this is an artificial delay, but this is the truth of a startup and oasis had already booted before the delay anyway)