“Static Linking Considered Harmful” Considered Harmful(gavinhoward.com)
gavinhoward.com
“Static Linking Considered Harmful” Considered Harmful
https://gavinhoward.com/2021/10/static-linking-considered-harmful-considered-harmful/
236 comments
While you are technically correct, ldd is literally a bash script which runs ld-linux.so...
ld-linux.so's job is to link and run an executable. It's not a vulnerability that it runs executable it's been handed. If you run "ld-linux.so /bin/ls" you're just running "ls" There's no security issue with this behavior. It's an interpreter like any other - same as /usr/bin/python.
The argument around ldd having a vulnerability is that it appears to be an introspective tool. It is not immediately obvious that it just executes "LD_TRACE_LOADED_OBJECTS=1 ld-linux.so /bin/ls" to print what the linker does during an actual, live linking operation.
ldd documenting this fact to remind people seems reasonable to me. There are other tools like readelf which can inspect objects without executing their contents. Dynamic linking is, well, dynamic and it can depend on code executed at runtime -- so it is necessary to do so to get an accurate report of what will happen.
The argument around ldd having a vulnerability is that it appears to be an introspective tool. It is not immediately obvious that it just executes "LD_TRACE_LOADED_OBJECTS=1 ld-linux.so /bin/ls" to print what the linker does during an actual, live linking operation.
ldd documenting this fact to remind people seems reasonable to me. There are other tools like readelf which can inspect objects without executing their contents. Dynamic linking is, well, dynamic and it can depend on code executed at runtime -- so it is necessary to do so to get an accurate report of what will happen.
Author here.
Thank you for pointing out my mistake. I'll fix it.
Thank you for pointing out my mistake. I'll fix it.
ld isn't the dynamic linker either, it's the object linker. the name of the dynamic linker is ld.so or ld-linux.so. this is also documented in the ldd man page.
Fun fact: despite the library path and name, ld-linux.so is executable. Try running
/lib64/ld-linux-x86-64.so.2
In fact, /lib64/ld-linux-x86-64.so.2 is set as the interpreter of the dynamic executables, so executing ./dyn_exec ends up being equivalent to /lib64/ld-linux-x86-64.so.2 ./dyn_exec. The explicit form is sometimes useful, for instace if you want to execute the file with an interpreter other than the one specified in the executable, or if you want to pass one of the parameters that ld.so accepts.
It's also useful if you accidentally chmod -x chmod...
Secure - and more useful - replacement: lddtree.
Part of "pax-utils" or similar package on some distros. It's actually a shell script wrapper around scanelf (also pax utils), which reads the appropriate bytes from ELF files without just executing the ELF file.
Part of "pax-utils" or similar package on some distros. It's actually a shell script wrapper around scanelf (also pax utils), which reads the appropriate bytes from ELF files without just executing the ELF file.
I usually just do: objdump -x <binary> | grep NEEDED but this is a great tip !
Often you need to find not only the soname but the actual .so file that is found and linked. This can be affected by environment variables like LD_PRELOAD and LD_LIBRARY_PATH. Unfortunately I don't know of a way to check this without ldd. If somebody knows of a tool that can do this without executing the binary, please share.
lddtree does handle LD_LIBRARY_PATH. It doesn't care about LD_PRELOAD though.
The first section, on security fixes ("[...] you only need to update a shared library to apply security fixes to all executables that rely on that library"), starts talking about security fixes but then suddenly switches to talking about ABI and API breaks. But that sleight-of-hand hides the fact that many (perhaps even most) security fixes do not break the ABI or API; they are completely contained to the implementation (one obvious exception would be if the security issue was caused by bad API design, but even then often there are ways to fix it without breaking the ABI).
> But that sleight-of-hand hides the fact that many (perhaps even most) security fixes do not break the ABI or API; they are completely contained to the implementation (one obvious exception would be if the security issue was caused by bad API design, but even then often there are ways to fix it without breaking the ABI).
Right you are. I was also perplexed when I read that non sequitur. The author's reference to DLL Hell also suggests there's some confusion in his analysis of the underlying problem, given that DLL Hell is very specific to windows and at best is only orthogonally related to ABI. The author's references to API changes make even less sense, and definitely cast doubt over his insight into the issue.
Right you are. I was also perplexed when I read that non sequitur. The author's reference to DLL Hell also suggests there's some confusion in his analysis of the underlying problem, given that DLL Hell is very specific to windows and at best is only orthogonally related to ABI. The author's references to API changes make even less sense, and definitely cast doubt over his insight into the issue.
And was kind of fixed with application manifests, on XP.
Differentiating DLL and SO hell is getting a bit beyond pedantic as they are implementations of the same fundamental abstraction. Any substantial difference in merely one of implementation details.
> Differentiating DLL and SO hell is getting a bit beyond pedantic (...)
It really isn't. Unlike linking problems, where the problem is focused on how you need to fight your dependencies to be able to resolve symbols, DLL Hell has been for over a decade a dependency resolution problem that is solved at the packaging level.
More importantly, on Windows, where sharing DLLs is not a thing, you can simply drop a DLL in the app dir and be done with it. In fact, it's customary for windows apps to just bundle all their dependencies.
It really isn't. Unlike linking problems, where the problem is focused on how you need to fight your dependencies to be able to resolve symbols, DLL Hell has been for over a decade a dependency resolution problem that is solved at the packaging level.
More importantly, on Windows, where sharing DLLs is not a thing, you can simply drop a DLL in the app dir and be done with it. In fact, it's customary for windows apps to just bundle all their dependencies.
Author here.
Sure, most security fixes do not break the ABI or API. But it only takes one, and then you have a security issue.
So I can understand your sentiment that it was sleight-of-hand, but I argue that it's not.
Also, we only think that most security updates don't break API or ABI because we have no automatic way of checking. It's all thoughts and prayers. I'm a religious person, but thoughts and prayers are not a good way to do computer security.
Sure, most security fixes do not break the ABI or API. But it only takes one, and then you have a security issue.
So I can understand your sentiment that it was sleight-of-hand, but I argue that it's not.
Also, we only think that most security updates don't break API or ABI because we have no automatic way of checking. It's all thoughts and prayers. I'm a religious person, but thoughts and prayers are not a good way to do computer security.
I've (probably?) found more linker vulnerabilities than anyone else, but don't really understand your argument. I definitely understand library search path vulnerabilities (I've literally found bugs in $ORIGIN parsing, e.g. CVE-2010-3847 was one of mine!).
We think security updates don't break API/ABI because they're backported to all previous major versions that are still actively supported. This isn't "thoughts and prayers", it's work that the packagers and maintainers do. I can't tell you how useful being able to fix a vulnerability in every program is, just think of updating every application that uses libpng on your desktop, or every server application that uses OpenSSL.
I just can't imagine wanting to give that up, because of name mangling ABI issues!
We think security updates don't break API/ABI because they're backported to all previous major versions that are still actively supported. This isn't "thoughts and prayers", it's work that the packagers and maintainers do. I can't tell you how useful being able to fix a vulnerability in every program is, just think of updating every application that uses libpng on your desktop, or every server application that uses OpenSSL.
I just can't imagine wanting to give that up, because of name mangling ABI issues!
I hope that puts it to rest; I can hope. Unfortunately, my experience is that people who haven't fixed multiple security problems, and probably have never heard of taviso, know a lot more about security than people who have...
> Sure, most security fixes do not break the ABI or API. But it only takes one, and then you have a security issue.
As someone who has been a Unix sysadmin for almost twenty years now, I don't recall every installing a patch or package update that broke the ABI/API.
If one should happen to occur next week, I'll still go with leaning towards linked libraries, as their convenience out-weights (has out-weighted) the once-in-twenty-years occurrence of breakage.
> It's all thoughts and prayers. I'm a religious person, but thoughts and prayers are not a good way to do computer security.
Ditto. But "faith" and "trust" are (almost) synonyms. And I trust the updaters of libraries to be able to get out a security update in short order more than I trust the possibility of all the package maintainers that link to it to be able to coördinate a quick update.
When OpenSSL's Heartbleed occurred it was one package update and restarting of a bunch of processes (helped by the checkrestart utility in the debian-goodies package). That's a lot quicker than updating a bazillion dependents.
As someone who has been a Unix sysadmin for almost twenty years now, I don't recall every installing a patch or package update that broke the ABI/API.
If one should happen to occur next week, I'll still go with leaning towards linked libraries, as their convenience out-weights (has out-weighted) the once-in-twenty-years occurrence of breakage.
> It's all thoughts and prayers. I'm a religious person, but thoughts and prayers are not a good way to do computer security.
Ditto. But "faith" and "trust" are (almost) synonyms. And I trust the updaters of libraries to be able to get out a security update in short order more than I trust the possibility of all the package maintainers that link to it to be able to coördinate a quick update.
When OpenSSL's Heartbleed occurred it was one package update and restarting of a bunch of processes (helped by the checkrestart utility in the debian-goodies package). That's a lot quicker than updating a bazillion dependents.
Not related to the OP but I’ve had many issues with non standard vendors placing breaking changes in their patch/minor releases.
You think you’re updating to a relatively normal version and it actually breaks your entire production.
I’m looking at you, MySQL, docker, Mongodb, or harbor.
I’m not complaining, those are provided free of charge and we are happy when it works, but semver is really all that matters in this: either you adhere and it’s smooth sailing, or you don’t and then please don’t use semver-looking numbering.
I’m looking at you, MySQL, docker, Mongodb, or harbor.
I’m not complaining, those are provided free of charge and we are happy when it works, but semver is really all that matters in this: either you adhere and it’s smooth sailing, or you don’t and then please don’t use semver-looking numbering.
There's a difference between vendor provided packages and distro provided packages. Distros pay a lot more attention to breaking changes.
[deleted]
I understand where you are coming from. That's actually why I gave my ideas to get the best from both types of linking.
That's why you run a trusted Linux distro in production, like sles or rhel or ubuntu, their core value proposition is _exactly_ this: Provide binary compatible drop in replacement security updates for shared objects over the lifetime of the platform. They even maintain kernel internal abi compatibility for the released and maintained product, so your exotic-hardware.ko module doesn't need to be updated.
Also there is mature tooling around ABI compliance checking: https://lvc.github.io/abi-compliance-checker/
This is used to document the ABI stability of mature projects like so: https://abi-laboratory.pro/index.php?view=tracker
E: re thoughts and prayers, the mature Linux distro all run extensive revalidation for their security updates. Breaking customer binary applications with a security update is the absolute worst case scenario. For yocto or similar cost free projects, or Linux distros that are source, not binary, that may be a different story. But for sles, rhel, ubuntu it's not thoughts and prayers but a maintenance verification and release process on their side and a monthly bill on yours.
Also there is mature tooling around ABI compliance checking: https://lvc.github.io/abi-compliance-checker/
This is used to document the ABI stability of mature projects like so: https://abi-laboratory.pro/index.php?view=tracker
E: re thoughts and prayers, the mature Linux distro all run extensive revalidation for their security updates. Breaking customer binary applications with a security update is the absolute worst case scenario. For yocto or similar cost free projects, or Linux distros that are source, not binary, that may be a different story. But for sles, rhel, ubuntu it's not thoughts and prayers but a maintenance verification and release process on their side and a monthly bill on yours.
If a security fix must break the API, you can bump up the SONAME which forces all packages depending on your shared library to recompile against the new version. This is comparable to the requirements under static linking, but you only have to do it when absolutely needed.
> we have no automatic way of checking
I do for Fedora packages I maintain. There can be false positives, like on symbols that shouldn't be exported, but you can look at the API to start with. (The library author should do that, of course.)
I do for Fedora packages I maintain. There can be false positives, like on symbols that shouldn't be exported, but you can look at the API to start with. (The library author should do that, of course.)
[deleted]
> I don’t think Ulrich Drepper could have foreseen all of the problems with DLL Hell (though there were signs on Windows),
Drepper's article is from 2006. "DLL Hell" was beyond fully understood at that point, certainly by Drepper and large numbers of other well informed programmers.
Drepper's article is from 2006. "DLL Hell" was beyond fully understood at that point, certainly by Drepper and large numbers of other well informed programmers.
I would even claim DLL didn't really exist anymore in 2006. I remember it from the late 90s, but haven't encountered it in the last 15 years. Fortunately, people have learned their lessons.
Author here.
In everything in the post, I attempted to give Drepper the benefit of the doubt, though there are cases where I don't want to since he sounds (to me) paternalistic in the way Apple and Microsoft do.
So perhaps he understood, but I didn't want to assume that, especially since it might inflame the discussion, which is exactly what I was trying to avoid doing (again).
In everything in the post, I attempted to give Drepper the benefit of the doubt, though there are cases where I don't want to since he sounds (to me) paternalistic in the way Apple and Microsoft do.
So perhaps he understood, but I didn't want to assume that, especially since it might inflame the discussion, which is exactly what I was trying to avoid doing (again).
DLL Hell was well understood in the mid-90s. I think you need to give him more benefit of the doubt on this point.
> DLL Hell was well understood in the mid-90s.
Also, there are some additional facets of DLL Hell which happened in the Windows of the mid-90s which are not as relevant to Linux. What made DLL Hell so bad there was that installing any random program could replace globally shared libraries, sometimes even with an older version. That is, installing a game could make an unrelated productivity app stop working, because the game helpfully installed newer (or older!) versions of the shared libraries it needed, into the same globally shared directory in which nearly all DLLs lived. It got so bad (even DLLs which came with the operating system were being overwritten) that Microsoft IIRC initially introduced a system which detected when this happened, and replaced the DLLs again with a clean copy it had stashed somewhere else (and later, made these files more directly protected).
That's before considering the disaster that is in-process COM servers; presenting a standard "open file" dialog, or doing some printing, is enough to make Windows load arbitrary DLLs into your process (shell extensions and/or printer drivers), and these often don't have the highest code quality. And then there are some things which inject arbitrary DLLs into every process...
Compared to that, the dynamic linking issues in the Linux world are much more bearable. You don't see arbitrary programs overwriting the global copy of something like libgtk or openssl or zlib or libc, the only arbitrary dynamic libraries being loaded into a process are things like the NSS ones (usually from a small well behaved set) or plugins from the graphics libraries (also usually from a small well behaved set), and the only dynamic library being injected into every process is the vDSO from the kernel.
Also, there are some additional facets of DLL Hell which happened in the Windows of the mid-90s which are not as relevant to Linux. What made DLL Hell so bad there was that installing any random program could replace globally shared libraries, sometimes even with an older version. That is, installing a game could make an unrelated productivity app stop working, because the game helpfully installed newer (or older!) versions of the shared libraries it needed, into the same globally shared directory in which nearly all DLLs lived. It got so bad (even DLLs which came with the operating system were being overwritten) that Microsoft IIRC initially introduced a system which detected when this happened, and replaced the DLLs again with a clean copy it had stashed somewhere else (and later, made these files more directly protected).
That's before considering the disaster that is in-process COM servers; presenting a standard "open file" dialog, or doing some printing, is enough to make Windows load arbitrary DLLs into your process (shell extensions and/or printer drivers), and these often don't have the highest code quality. And then there are some things which inject arbitrary DLLs into every process...
Compared to that, the dynamic linking issues in the Linux world are much more bearable. You don't see arbitrary programs overwriting the global copy of something like libgtk or openssl or zlib or libc, the only arbitrary dynamic libraries being loaded into a process are things like the NSS ones (usually from a small well behaved set) or plugins from the graphics libraries (also usually from a small well behaved set), and the only dynamic library being injected into every process is the vDSO from the kernel.
If someone glosses over an issue, it's either innocent (didn't know about it, didn't understand it) or deceptive (intentionally ignored it). Giving Drepper the benefit of the doubt here means assuming he didn't know or understand the problem he failed to adequately address.
Here is most of the relevant text from TFA:
> Ulrich Drepper claims, rightly, that you only need to update a shared library to apply security fixes to all executables that rely on that library.
> This is a good thing! It is also a good vision.
> Unfortunately, it is only a vision because it’s not the whole story.
> As of yet, there is no way to automatically determine if the ABI or API of a shared library changed between updates. This is called DLL Hell.
The final two words are a link to the wikipedia page on "DLL Hell". The final sentence of the intro section there states:
> DLL Hell is the Windows ecosystem-specific form of the general concept dependency hell.
That is, TFA defines "DLL Hell" using a wikipedia page that explicitly states that it is a Windows-specific version of a more general problem.
Drepper's article from 2006 fully tackles the general problem, albeit without (as the TFA puts it): "a way to automatically determine if the ABI or API of a shared library changed between updates." Drepper's primary suggestion there is a naming/versioning scheme which describes specifically whether the ABI/API has changed in ways that matter for shared linkage. It's not automatic - that part is true. But is is a solution, widely used in Linux and *nix more broadly.
Did Drepper address the Windows specific parts of "DLL Hell". He did not, but that's because "DLL Hell" is Windows specific (as other comments here fully clarify), and he was not writing about how to fix the full scope of that particular nightmare. He likely understood "DLL Hell" as well as anyone, nevertheless.
> Ulrich Drepper claims, rightly, that you only need to update a shared library to apply security fixes to all executables that rely on that library.
> This is a good thing! It is also a good vision.
> Unfortunately, it is only a vision because it’s not the whole story.
> As of yet, there is no way to automatically determine if the ABI or API of a shared library changed between updates. This is called DLL Hell.
The final two words are a link to the wikipedia page on "DLL Hell". The final sentence of the intro section there states:
> DLL Hell is the Windows ecosystem-specific form of the general concept dependency hell.
That is, TFA defines "DLL Hell" using a wikipedia page that explicitly states that it is a Windows-specific version of a more general problem.
Drepper's article from 2006 fully tackles the general problem, albeit without (as the TFA puts it): "a way to automatically determine if the ABI or API of a shared library changed between updates." Drepper's primary suggestion there is a naming/versioning scheme which describes specifically whether the ABI/API has changed in ways that matter for shared linkage. It's not automatic - that part is true. But is is a solution, widely used in Linux and *nix more broadly.
Did Drepper address the Windows specific parts of "DLL Hell". He did not, but that's because "DLL Hell" is Windows specific (as other comments here fully clarify), and he was not writing about how to fix the full scope of that particular nightmare. He likely understood "DLL Hell" as well as anyone, nevertheless.
I think you're missing the point. When the only alternatives are intellectual dishonesty or ignorance, assuming ignorance is the more generous option. Therefore, assuming he didn't know is giving him the benefit of the doubt.
The reasoning behind your differing assumption is beside the point, even though your reasoning is more likely to be true.
The reasoning behind your differing assumption is beside the point, even though your reasoning is more likely to be true.
I think you're missing my point.
The TFA's claim is that Drepper didn't get all the issues with DLL Hell. My claim is that Drepper did get all of the issues, and addressed those that were not Windows-specific (e.g. per-process COM servers etc.)
The TFA's claim is that Drepper didn't get all the issues with DLL Hell. My claim is that Drepper did get all of the issues, and addressed those that were not Windows-specific (e.g. per-process COM servers etc.)
No. I am not addressing your point because your point is not logically related to the topic under discussion.
Specifically, your post said:
> DLL Hell was well understood in the mid-90s. I think you need to give him more benefit of the doubt on this point.
The substantive question about whether he adequately addressed the issues with dynamic linking is beside the point. Feel free to agree or disagree with the above poster. Not relevant.
The post I am responding to is about, given ghoward's position that X did not adequately address topic Y, whether it is "giving X the benefit of the doubt" to further assume (a) X did know about topic Y; or (b) X did not know about topic Y.
Because (a) implies dishonesty but (b) merely implies ignorance, "giving X the benefit of the doubt" means you should assume (b) until proven otherwise.
If you want to respond to other topics, feel free to hit the reply button below someone else's post.
Specifically, your post said:
> DLL Hell was well understood in the mid-90s. I think you need to give him more benefit of the doubt on this point.
The substantive question about whether he adequately addressed the issues with dynamic linking is beside the point. Feel free to agree or disagree with the above poster. Not relevant.
The post I am responding to is about, given ghoward's position that X did not adequately address topic Y, whether it is "giving X the benefit of the doubt" to further assume (a) X did know about topic Y; or (b) X did not know about topic Y.
Because (a) implies dishonesty but (b) merely implies ignorance, "giving X the benefit of the doubt" means you should assume (b) until proven otherwise.
If you want to respond to other topics, feel free to hit the reply button below someone else's post.
The point under discussion is this line (and the surrounding text) from TFA:
> I don’t think Ulrich Drepper could have foreseen all of the problems with DLL Hell (though there were signs on Windows),
> I don’t think Ulrich Drepper could have foreseen all of the problems with DLL Hell (though there were signs on Windows),
I'm pretty sure I know what I replied to, and it wasn't that. If you want to go get into some grary internet argument with someone else about that, feel free, but it's off-topic in reply to my comment. Thanks.
Bit weird to complain about that when you commented under their comment: https://news.ycombinator.com/item?id=28737096
Come again? How, exactly?
I replied to their comment, and substantively responded to something that was actually in their comment.
How is that inconsistent?
I replied to their comment, and substantively responded to something that was actually in their comment.
How is that inconsistent?
[deleted]
[deleted]
I'm having a hard time understanding because if DLL Hell was well understood by 2006, it seems I should have given him less benefit of the doubt, but you say I should give him more.
Apologies.
Apologies.
Considered harmful essays considered harmful:
https://meyerweb.com/eric/comment/chech.html
https://meyerweb.com/eric/comment/chech.html
Author here.
I read that essay a while ago and reread it before publishing my post.
I don't think the author makes a good case. They make bare assertions without examples and without evidence, so it's unconvincing.
For example, they say that a "Considered Harmful" essay:
> often serves to inflame whatever debate is in progress, and thus makes it that much harder for a solution to be found through any means.
But this could be true of any essay.
More to the point, I think that the tone of the essay or post matters more. Case in point, my "Dynamic Linking Needs to Die" post probably inflames the debate more than this new "Considered Harmful" post.
Maybe "Considered Harmful" essays have a reputation for inflaming the debate, and there's an argument to be made for that. But I personally don't think my post is one that follows that reputation.
Also, because I was writing a post about a pre-existing "Considered Harmful" essay, it was natural to use it, especially since I needed to make it clear that my previous post is harmful too.
I read that essay a while ago and reread it before publishing my post.
I don't think the author makes a good case. They make bare assertions without examples and without evidence, so it's unconvincing.
For example, they say that a "Considered Harmful" essay:
> often serves to inflame whatever debate is in progress, and thus makes it that much harder for a solution to be found through any means.
But this could be true of any essay.
More to the point, I think that the tone of the essay or post matters more. Case in point, my "Dynamic Linking Needs to Die" post probably inflames the debate more than this new "Considered Harmful" post.
Maybe "Considered Harmful" essays have a reputation for inflaming the debate, and there's an argument to be made for that. But I personally don't think my post is one that follows that reputation.
Also, because I was writing a post about a pre-existing "Considered Harmful" essay, it was natural to use it, especially since I needed to make it clear that my previous post is harmful too.
>Maybe "Considered Harmful" essays have a reputation for inflaming the debate, and there's an argument to be made for that. But I personally don't think my post is one that follows that reputation.
Neither your post title nor your content bothered me but just as fyi... some people are very irritated by "snowclones". My previous comment with various examples: https://news.ycombinator.com/item?id=19071532
Neither your post title nor your content bothered me but just as fyi... some people are very irritated by "snowclones". My previous comment with various examples: https://news.ycombinator.com/item?id=19071532
Yeah, I suspected people might be. But since I was answering a "Considered Harmful" essay, it was too easy. Eh, oh well.
Either way, "considered harmful" is a tired and unoriginal cliché. Using it twice in one title is just stonkingly uncreative.
I agree with most of the article but the title sucks!
Also I'm not sure how you wrote so much about using LLVM IR as a portable executable format without mentioning WebAssembly or PNaCl once!
I agree with most of the article but the title sucks!
Also I'm not sure how you wrote so much about using LLVM IR as a portable executable format without mentioning WebAssembly or PNaCl once!
> Either way, "considered harmful" is a tired and unoriginal cliché. Using it twice in one title is just stonkingly uncreative.
> I agree with most of the article but the title sucks!
You are correct. I don't claim to be creative, especially with post titles. To be honest, I'm surprised this post got as much traction as it did.
> Also I'm not sure how you wrote so much about using LLVM IR as a portable executable format without mentioning WebAssembly or PNaCl once!
Because, in my opinion, they will not solve the problem.
WebAssembly is too constrained.
PNaCl uses LLVM, which would need to be redesigned to do all that I want.
> I agree with most of the article but the title sucks!
You are correct. I don't claim to be creative, especially with post titles. To be honest, I'm surprised this post got as much traction as it did.
> Also I'm not sure how you wrote so much about using LLVM IR as a portable executable format without mentioning WebAssembly or PNaCl once!
Because, in my opinion, they will not solve the problem.
WebAssembly is too constrained.
PNaCl uses LLVM, which would need to be redesigned to do all that I want.
I was wondering when something like this would be written. We’ve come full circle.
I read this interesting article recently about all the security problems that static linking, bundling and pinning cause for Linux distributions:
https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag...
https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag...
Author here, and I am also a Gentoo user on my daily driver.
I read that article when it came out, and it felt disingenuous.
mgorny helps maintain Gentoo, which of all distributions, does rebuild software as necessary.
It would be a simple change to portage to rebuild all dependencies of a library when that library is updated, regardless of if the user uses static linking or not. In fact, there is a project to have a statically-linked musl-based Gentoo.
So it's possible and (I would argue) would be an easy change. mgorny would not have to worry about much because he could, in essence, treat all libraries as dynamic libraries. The only pain to be felt would be by users who would see their build times increase (to rebuild dependents).
As a user, I would happily take that tradeoff. I already build my browser, which takes 5-8 hours; I'll happily rebuild dependents of updated libraries.
I read that article when it came out, and it felt disingenuous.
mgorny helps maintain Gentoo, which of all distributions, does rebuild software as necessary.
It would be a simple change to portage to rebuild all dependencies of a library when that library is updated, regardless of if the user uses static linking or not. In fact, there is a project to have a statically-linked musl-based Gentoo.
So it's possible and (I would argue) would be an easy change. mgorny would not have to worry about much because he could, in essence, treat all libraries as dynamic libraries. The only pain to be felt would be by users who would see their build times increase (to rebuild dependents).
As a user, I would happily take that tradeoff. I already build my browser, which takes 5-8 hours; I'll happily rebuild dependents of updated libraries.
> I already build my browser, which takes 5-8 hours; I'll happily rebuild dependents of updated libraries
This is ridiculous. I most definitely do NOT want to rebuild literally EVERY SINGLE PACKAGE on my system the moment they fix a typo in libz.
With static linking literally every other Gentoo upgrade becomes a world rebuild. See Nix. It is just insane and the main reason I don't use NixOS (without a 3rd party binary cache/host) for anything other than very small systems.
This is ridiculous. I most definitely do NOT want to rebuild literally EVERY SINGLE PACKAGE on my system the moment they fix a typo in libz.
With static linking literally every other Gentoo upgrade becomes a world rebuild. See Nix. It is just insane and the main reason I don't use NixOS (without a 3rd party binary cache/host) for anything other than very small systems.
> This is ridiculous. I most definitely do NOT want to rebuild literally EVERY SINGLE PACKAGE on my system the moment they fix a typo in libz.
I understand. You don't. I do.
What my post is arguing for is not one or the other, but giving the user the choice.
I think you would be okay with making the choice of not rebuilding everything, and I would make the opposite choice.
I want to live in a world where we can both choose.
I understand. You don't. I do.
What my post is arguing for is not one or the other, but giving the user the choice.
I think you would be okay with making the choice of not rebuilding everything, and I would make the opposite choice.
I want to live in a world where we can both choose.
But you can already rebuild the entire world on every package change if you want (for sadomasochistic reasons? it basically makes gentoo with say KDE _unusable_); and I fail to see if there would be any practical difference on whether you statically link, you dynamically link to a hash-based soname a la nix, or you dynamically link -- you are just rebuilding everything on every library change, and you are already assuming ASLR is useless.
> It would be a simple change to portage to rebuild all dependencies of a library when that library is updated
I believe e.g. freebsd's portmaster takes a -r switch to mean "rebuild this and all its dependents" (I presume you meant revdeps here), which gets you quite a long way.
I believe e.g. freebsd's portmaster takes a -r switch to mean "rebuild this and all its dependents" (I presume you meant revdeps here), which gets you quite a long way.
> It would be a simple change to portage to rebuild all dependencies of a library when that library is updated
No change needed. It can already be done with the special ":=" dependency slot: https://devmanual.gentoo.org/general-concepts/dependencies/#...
No change needed. It can already be done with the special ":=" dependency slot: https://devmanual.gentoo.org/general-concepts/dependencies/#...
But should distributing and ensuring the security of software that users install on their own systems even be the responsibility of operating system maintainers?
Isn't it obvious? Or do you suggest that end users should be responsible for checking and patching every package on their system? Given that Red Hat has a large department just to do that for RHEL, I can't imagine where end users would begin.
No, I’m suggesting that the authors of the applications are responsible for patching and distributing their own application like on every other major OS.
There are examples of the sort of reason I want dynamic linking under https://news.ycombinator.com/item?id=28734875 and you also really want it for profiling/tracing, and possibly debugging, e.g. in HPC with the MPI PMPI mechanism.
libabigail is one tool for checking that a library's ELF versioning hasn't lied about ABI compatibility when you're building packages, for instance.
Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc.
libabigail is one tool for checking that a library's ELF versioning hasn't lied about ABI compatibility when you're building packages, for instance.
Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc.
Author here.
Yeah, there are reasons to use dynamic linking, but I'm still not sure why dynamic linking gives you better profiling and tracing.
> Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc.
I mention this in the post, but I quoted another person, Ian Lance Taylor, who wrote the gold linker, as saying that symbol combining, a portion of symbol versioning, is an unsolved problem.
Even if developers don't follow Drepper's advice, I wouldn't put the blame on them because of that fact; I would argue that it's more accurate to say they can't really follow his advice.
Yeah, there are reasons to use dynamic linking, but I'm still not sure why dynamic linking gives you better profiling and tracing.
> Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc.
I mention this in the post, but I quoted another person, Ian Lance Taylor, who wrote the gold linker, as saying that symbol combining, a portion of symbol versioning, is an unsolved problem.
Even if developers don't follow Drepper's advice, I wouldn't put the blame on them because of that fact; I would argue that it's more accurate to say they can't really follow his advice.
> Yeah, there are reasons to use dynamic linking, but I'm still not sure why dynamic linking gives you better profiling and tracing.
It's not so much that the tracing becomes better, but that it becomes feasible at all. Two specific situations come to mind, both MPI-adjacent. (1) Running a PMPI-based tool on code you can't recompile yourself (e.g., you need a Q clearance to see the source, but not to actually execute it--weird I know, but not that uncommon in the DOE labs.); and (2) running multiple PMPI-based tools simultaneously which are composed at runtime via PnMPI.
It's not so much that the tracing becomes better, but that it becomes feasible at all. Two specific situations come to mind, both MPI-adjacent. (1) Running a PMPI-based tool on code you can't recompile yourself (e.g., you need a Q clearance to see the source, but not to actually execute it--weird I know, but not that uncommon in the DOE labs.); and (2) running multiple PMPI-based tools simultaneously which are composed at runtime via PnMPI.
Exactly, but even if you can rebuild it, you don't want to under most circumstances. In principle with static binaries you can use dyninst, for instance, but in practice you may not be able to for various reasons.
Then as a system manager you'd like to have global profiling of what runs -- for various reasons, including putting the result for a program in front of a user. If it's all dynamically linked, you can hook in and do that by default, and with various levels of insistence, depending on the hooks.
I addressed these arguments. I think that if you don't have access to the source, it's a security problem.
That includes not having access to the source because of security clearances. As far as I am concerned, your superiors in the DOE are adversaries to you and your machine.
Other people will have different viewpoints on that, and that's fine.
That includes not having access to the source because of security clearances. As far as I am concerned, your superiors in the DOE are adversaries to you and your machine.
Other people will have different viewpoints on that, and that's fine.
I want free software, and I'm aghast at what some people run, but this is not the real world. How many examples do you want? If you talk security, what's the threat model? What I run on a decently managed (with the aid of dynamically linked libraries) compute cluster should only put my data at risk.
As I understood it, there actually has been a push for free software solutions on CORAL systems, but I don't remember where that came from.
As I understood it, there actually has been a push for free software solutions on CORAL systems, but I don't remember where that came from.
Drepper's scheme obviously works tolerably well with glibc, in particular, even if he didn't know what he was talking about.
If you don't have symbol versioning, and you want to statically link something with parts that depend on incompatible versions of a library, you're obviously stuffed.
If you don't have symbol versioning, and you want to statically link something with parts that depend on incompatible versions of a library, you're obviously stuffed.
My personal view is that only standard core system libraries should be dynamically linked. Anything that is not totally ubiquitous should be static.
That seems to be the Go and Rust default approach, or close to it. Link libc and such, but build in more obscure things.
The idea of dynamically linking everything doesn’t scale. It asks too much of distribution maintainers and makes them the choke point for every little upgrade.
That seems to be the Go and Rust default approach, or close to it. Link libc and such, but build in more obscure things.
The idea of dynamically linking everything doesn’t scale. It asks too much of distribution maintainers and makes them the choke point for every little upgrade.
> That seems to be the Go and Rust default approach
I think Go tried to go fully static but ran into problems as most mainstream OSes except for Linux do not provide a stable system call interface, so applications have to link at least one dynamic library. The Linux system call interface is stable, but expects a C style stack layout and that also bit the Go devs. multiple times. So you can probably go fully static if you are only targeting Linux.
I think Go tried to go fully static but ran into problems as most mainstream OSes except for Linux do not provide a stable system call interface, so applications have to link at least one dynamic library. The Linux system call interface is stable, but expects a C style stack layout and that also bit the Go devs. multiple times. So you can probably go fully static if you are only targeting Linux.
> I think Go tried to go fully static but ran into problems [...] So you can probably go fully static if you are only targeting Linux.
There's yet another problem Go ran into on Linux: resolving hostnames. The official way to resolve hostnames in a glibc-based system is to read /etc/nsswitch.conf, dynamically load the libraries referenced there, and call each of them in the correct order. Go tried to emulate that mechanism, going through glibc only when it saw a NSS module it didn't know, but that caused issues on musl-based systems which used a different mechanism (https://github.com/golang/go/issues/35305). And it cannot ever be fully static on glibc-based systems, since there's always the chance of someone adding a new module to /etc/nsswitch.conf (for instance, some Linux distributions have now added the newer "resolve" module to that line, so Go would have to implement yet another one to keep being fully static).
The Rust approach of dynamically linking to the shared C libraries (statically linking only the Rust libraries) avoided all these issues.
There's yet another problem Go ran into on Linux: resolving hostnames. The official way to resolve hostnames in a glibc-based system is to read /etc/nsswitch.conf, dynamically load the libraries referenced there, and call each of them in the correct order. Go tried to emulate that mechanism, going through glibc only when it saw a NSS module it didn't know, but that caused issues on musl-based systems which used a different mechanism (https://github.com/golang/go/issues/35305). And it cannot ever be fully static on glibc-based systems, since there's always the chance of someone adding a new module to /etc/nsswitch.conf (for instance, some Linux distributions have now added the newer "resolve" module to that line, so Go would have to implement yet another one to keep being fully static).
The Rust approach of dynamically linking to the shared C libraries (statically linking only the Rust libraries) avoided all these issues.
So, you link against a GUI toolkit that can be used to display images. Images in many different formats, including some that haven't been used in more than a decade and are unlikely to ever be encountered.
Do you want the GUI toolkit linkage to automatically include every object module for every possible image file format in your static image, or do you want the GUI toolkit to use runtime dynamic linkage (explicit or implicit) to import only the modules required for the image formats actually encountered?
Do you want the GUI toolkit linkage to automatically include every object module for every possible image file format in your static image, or do you want the GUI toolkit to use runtime dynamic linkage (explicit or implicit) to import only the modules required for the image formats actually encountered?
If I'm distributing binaries as a commercial product, I definitely want the former.
What about formats that don't exist yet? What about giving you the ability to supply your own encoder/decoder for some niche format you're using?
There are many "commercial products" that run on Linux whose installation includes various *.so files that will be dynamically linked at run time. Are you saying this is a bad idea?
I'd like to see solutions like SDL2's become more widespread: https://old.reddit.com/r/linux_gaming/comments/1upn39/sdl2_a... At runtime, the first call into SDL sets up a jump table for all SDL functions afterwards. When SDL is statically linked, it uses those static functions by default, but it's possible to specify an environmental variable pointing at an SDL dynamic lib, and the jump table will use those functions instead.
According to Linus, "Shared libraries are not a good thing in general".
https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...
https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...
Author here.
I quote that email twice in my post. I decided not to quote that part of the email just in case it might inflame the debate.
But yes, he did say that.
I quote that email twice in my post. I decided not to quote that part of the email just in case it might inflame the debate.
But yes, he did say that.
I sorta think that static linking needs to die, but dynamic linking also needs to be a lot better.
More of the choices that get made during autoconf needs to be made lazily at link time. That would make executables and dynamic libraries more like containers or lego blocks that could be rearranged without compilation. Which would fix a lot of issues that people hate about package managers and bloated distros along the way.
More of the choices that get made during autoconf needs to be made lazily at link time. That would make executables and dynamic libraries more like containers or lego blocks that could be rearranged without compilation. Which would fix a lot of issues that people hate about package managers and bloated distros along the way.
Author here. I don't know why my own submission of this [1] didn't catch and this one did. Oh well. XD
I'm open to answering questions, and I'll be hanging in the thread.
[1]: https://news.ycombinator.com/item?id=28728801
I'm open to answering questions, and I'll be hanging in the thread.
[1]: https://news.ycombinator.com/item?id=28728801
I am extremely skeptical of this calculation by Drew DeVault:
> On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. A good linker will remove unused symbols.
Percent of exported symbols used is a terrible proxy for percent of library functionality used.
As an example, an SDL hello world program [1] uses 3 functions out of 750 exported by libSDL2. Does that mean it only uses 3/750 = 0.4% of libSDL2's functionality? Of course not; when I tried compiling it against a static libSDL2 instead of dynamic, the executable's binary size increased by a full 70% of the size of libSDL2.so. [2]
Now, on one hand I admit that's an extreme example, as a real SDL program would use a higher number of functions. And SDL itself is not very amenable to dead code stripping; a good chunk of the hello-world program is for things like sound or input that the it doesn't actually use.
Still, even the hello-world program legitimately needs a bunch of functionality, including support for both X11 and Wayland, and setting up hardware accelerated surfaces using OpenGL. And a real program would need more. Also, I'm not even counting the long list of shared libraries linked by SDL (ranging from libX11 to libFLAC); a SDL-based program will use 0% of their symbols (since it doesn't them directly), but may need a substantial portion of their functionality.
More importantly, this general pattern, where exported APIs are just the tip of the iceberg, applies to many libraries.
I tried a similar experiment with zpipe, the official example program for zlib – a much smaller and shallower library. It ended up using 23% of symbols but 53% of the implementation.
On the other end of the spectrum, the problem would be far worse with a real GUI library, where even, say, a simple text input box truly requires massive amounts of functionality including display, layout, Unicode font rendering, menus, internationalization, and keyboard and mouse input. Plus, of course, a variety of input method editors if you want to support languages like Japanese.
In a statically linked world you would probably want to implement most of that as IPC to shared daemons. Such a design could work; there isn't much that really needs the performance benefits of being in-process. But no major GUI libraries are written that way, because in a world with shared libraries there's no need.
[1] https://gist.github.com/comex/c64b5a7e409d5d48ad6ebf37ec068b...
[2] Test harness also at [1]. This is with LTO; without LTO the percentage is much higher. I also had to disable SDL's "dynapi" feature that makes static linking secretly actually dynamically link. Which they put in for good reasons, albeit mostly applicable only to proprietary software where the user can't just recompile/relink.
> On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. A good linker will remove unused symbols.
Percent of exported symbols used is a terrible proxy for percent of library functionality used.
As an example, an SDL hello world program [1] uses 3 functions out of 750 exported by libSDL2. Does that mean it only uses 3/750 = 0.4% of libSDL2's functionality? Of course not; when I tried compiling it against a static libSDL2 instead of dynamic, the executable's binary size increased by a full 70% of the size of libSDL2.so. [2]
Now, on one hand I admit that's an extreme example, as a real SDL program would use a higher number of functions. And SDL itself is not very amenable to dead code stripping; a good chunk of the hello-world program is for things like sound or input that the it doesn't actually use.
Still, even the hello-world program legitimately needs a bunch of functionality, including support for both X11 and Wayland, and setting up hardware accelerated surfaces using OpenGL. And a real program would need more. Also, I'm not even counting the long list of shared libraries linked by SDL (ranging from libX11 to libFLAC); a SDL-based program will use 0% of their symbols (since it doesn't them directly), but may need a substantial portion of their functionality.
More importantly, this general pattern, where exported APIs are just the tip of the iceberg, applies to many libraries.
I tried a similar experiment with zpipe, the official example program for zlib – a much smaller and shallower library. It ended up using 23% of symbols but 53% of the implementation.
On the other end of the spectrum, the problem would be far worse with a real GUI library, where even, say, a simple text input box truly requires massive amounts of functionality including display, layout, Unicode font rendering, menus, internationalization, and keyboard and mouse input. Plus, of course, a variety of input method editors if you want to support languages like Japanese.
In a statically linked world you would probably want to implement most of that as IPC to shared daemons. Such a design could work; there isn't much that really needs the performance benefits of being in-process. But no major GUI libraries are written that way, because in a world with shared libraries there's no need.
[1] https://gist.github.com/comex/c64b5a7e409d5d48ad6ebf37ec068b...
[2] Test harness also at [1]. This is with LTO; without LTO the percentage is much higher. I also had to disable SDL's "dynapi" feature that makes static linking secretly actually dynamically link. Which they put in for good reasons, albeit mostly applicable only to proprietary software where the user can't just recompile/relink.
Every time I read one of these essays extolling the virtues of static linking it makes me pretty sad. Engineering is about trade offs, and sometimes it does make sense to statically link, but the fact that those trade offs are so lopsided on Linux has very little to do with dynamic linking vs static linking as generic concepts, and more to do with the fact that the design of ld.so is straight out of the 1990s and almost nothing has be done to either exploit the benefits dynamic linking brings, nor to mitigate the issues it causes.
On Darwin derived systems (macOS, iOS, tvOS, watchOS) we have invested heavily in dynamic linking over the last two decades. That includes features we use to improve binary compatibility (things like two level namespaces (aka Direct Binding in ELF) and umbrella frameworks), middleware distribution (through techniques like bundling resources with their dylibs into frameworks), and mitigate the security issues (through technologies like __DATA_CONST).
Meanwhile, we also substantially reduced the cost of dynamic linking through things like the dyld shared cache (which makes dynamic linking of most system frameworks almost free, and in practice it often reduces their startup cost to below the startup cost of statically linking them once you include the cost of rebasing for PIE), and mitigate much of the rest of the cost of dynamic linking through things like pre-calculated launch closures. It does not hurt that my team owns both the static and dynamic linkers. We have a tight design loop so that when we come up with ideas for how to change libraries to make the dynamic linker load them faster we have the ability to rapidly deploy those changes through the ecosystem.
As I explained in here[1] dynamic linking on macOS is way faster than on Linux because we do it completely differently, and because of that it is used way more pervasively on our systems (a typical command line binary loads 80-150 dylibs, a GUI app around 300-800 depending on which of our OS you are talking about). IOW, by mitigating the costs we drove up the adoption which amplifies the benefits.
And it is not like we have squeezed all the performance out of the dynamic linker that we can, not by a long shot. We have more ideas than we have time to pursue, as well as additional improvements to static linking. If you have any interest in either static or dynamic linking we're looking for people to work on both[2].
[1]: https://www.realworldtech.com/forum/?threadid=197081&curpost...
[2]: https://jobs.apple.com/en-us/details/200235669/systems-engin...
(edit: fixed link formatting)
On Darwin derived systems (macOS, iOS, tvOS, watchOS) we have invested heavily in dynamic linking over the last two decades. That includes features we use to improve binary compatibility (things like two level namespaces (aka Direct Binding in ELF) and umbrella frameworks), middleware distribution (through techniques like bundling resources with their dylibs into frameworks), and mitigate the security issues (through technologies like __DATA_CONST).
Meanwhile, we also substantially reduced the cost of dynamic linking through things like the dyld shared cache (which makes dynamic linking of most system frameworks almost free, and in practice it often reduces their startup cost to below the startup cost of statically linking them once you include the cost of rebasing for PIE), and mitigate much of the rest of the cost of dynamic linking through things like pre-calculated launch closures. It does not hurt that my team owns both the static and dynamic linkers. We have a tight design loop so that when we come up with ideas for how to change libraries to make the dynamic linker load them faster we have the ability to rapidly deploy those changes through the ecosystem.
As I explained in here[1] dynamic linking on macOS is way faster than on Linux because we do it completely differently, and because of that it is used way more pervasively on our systems (a typical command line binary loads 80-150 dylibs, a GUI app around 300-800 depending on which of our OS you are talking about). IOW, by mitigating the costs we drove up the adoption which amplifies the benefits.
And it is not like we have squeezed all the performance out of the dynamic linker that we can, not by a long shot. We have more ideas than we have time to pursue, as well as additional improvements to static linking. If you have any interest in either static or dynamic linking we're looking for people to work on both[2].
[1]: https://www.realworldtech.com/forum/?threadid=197081&curpost...
[2]: https://jobs.apple.com/en-us/details/200235669/systems-engin...
(edit: fixed link formatting)
Author here.
I'm curious: did you read to the end of the post where I lay out ideas about how to get the benefits of both in one?
I'm curious: did you read to the end of the post where I lay out ideas about how to get the benefits of both in one?
Yes. They are all reasonable ideas, and we actually have experience with most (all?) of them:
* Distributing IR applications: We do a form of this by supporting bitcode for App Store submissions on iOS, tvOS, and watchOS. For watchOS it was a great success in the sense that it allowed us to transparently migrate all the binaries from armv7k (a 32 bit ABI on a 32 bit instruction set) to arm64_32 (a 32 bit on a 64 bit instruction set), but we had to carefully design both ABIs in parallel in order to allow that to be efficient. It also introduces serious burdens on developer workflows like crash reporting. Those probably are not significant issues for people deploying binaries to servers, but it can be pretty difficult for developers trying to aggregate crash statistics from apps deployed to consumer devices. It also causes security issues with code provenance since you have to accept locally signed code, have the transforms performed by a centrally trusted entity who signs it, or you have to limit to your optimizations to things you can verify through a provable chain back to the IR.
* Split stacks: We don't do this per se, but we do something semantically equivalent. When we designed the ABI from arm64e we used PAC (pointer authentication codes) to sign the return addresses on the stack, which means that while you can smash the stack and overwrite the return pointer, the authentication code won't match any more and you will crash.
* Pre-mapped libraries: We have done this since then Mac OS X Developer Releases back in the later 90s, though the mechanism has changed a number of times over the years. Originally we manually pre-assigned the address ranges of every dylib in the system (and in fact we carefully placed large gaps between the __TEXT and __DATA of the library and have the libraries in overlapping ranges so that we could place all the system __TEXT in single adjacent region and all the __DATA in a second adjacent region that way we could exploit the batch address translation registers on PPC processors to avoid polluting the TLBs). When the static linker built a dylib we look in a file with the mappings and build the dylib with segment base addresses that we looked up from the that manually maintained list, and when the OS booted we would pre-map all the libraries into their slots so every process just had them mapped.
That was a huge pain in the neck to maintain, since it meant whenever a library grew too large it was would break the optimizations and someone would need update the file and rebuild the OS. It also only dealt with rebasing, to deal with binding we locally ran update_prebinding which users hated because it was slow, sysadmins hated because it means we routinely rewrote every binary on the system. That was also before anyone had deployed ASLR or codesigning.
Nowadays we use the dyld shared cache to achieve similar ends, but it is a far more flexible mechanism. We essentially merge almost all the system dynamic libraries at OS build time into one mega-dylib that we can pre-bind together and sign. We also use VM tricks to allow the system to rebase it on page in rather than the dynamic linker doing any work, and we pre-map it into every address space by default.
We even perform a number of additional optimizations when we build it, such as analyzing the dependencies of every executable in the base system and pre-calculating a lot of the work dyld and even the dynamic runtimes like ObjC would have to do in order to avoid doing them during app launch.
So in short, I think your ideas there have more merit than you perhaps suspect, and from experience I can say if/when you implement them you implement them you might find your views on the trade offs of dynamic vs static linking change.
* Distributing IR applications: We do a form of this by supporting bitcode for App Store submissions on iOS, tvOS, and watchOS. For watchOS it was a great success in the sense that it allowed us to transparently migrate all the binaries from armv7k (a 32 bit ABI on a 32 bit instruction set) to arm64_32 (a 32 bit on a 64 bit instruction set), but we had to carefully design both ABIs in parallel in order to allow that to be efficient. It also introduces serious burdens on developer workflows like crash reporting. Those probably are not significant issues for people deploying binaries to servers, but it can be pretty difficult for developers trying to aggregate crash statistics from apps deployed to consumer devices. It also causes security issues with code provenance since you have to accept locally signed code, have the transforms performed by a centrally trusted entity who signs it, or you have to limit to your optimizations to things you can verify through a provable chain back to the IR.
* Split stacks: We don't do this per se, but we do something semantically equivalent. When we designed the ABI from arm64e we used PAC (pointer authentication codes) to sign the return addresses on the stack, which means that while you can smash the stack and overwrite the return pointer, the authentication code won't match any more and you will crash.
* Pre-mapped libraries: We have done this since then Mac OS X Developer Releases back in the later 90s, though the mechanism has changed a number of times over the years. Originally we manually pre-assigned the address ranges of every dylib in the system (and in fact we carefully placed large gaps between the __TEXT and __DATA of the library and have the libraries in overlapping ranges so that we could place all the system __TEXT in single adjacent region and all the __DATA in a second adjacent region that way we could exploit the batch address translation registers on PPC processors to avoid polluting the TLBs). When the static linker built a dylib we look in a file with the mappings and build the dylib with segment base addresses that we looked up from the that manually maintained list, and when the OS booted we would pre-map all the libraries into their slots so every process just had them mapped.
That was a huge pain in the neck to maintain, since it meant whenever a library grew too large it was would break the optimizations and someone would need update the file and rebuild the OS. It also only dealt with rebasing, to deal with binding we locally ran update_prebinding which users hated because it was slow, sysadmins hated because it means we routinely rewrote every binary on the system. That was also before anyone had deployed ASLR or codesigning.
Nowadays we use the dyld shared cache to achieve similar ends, but it is a far more flexible mechanism. We essentially merge almost all the system dynamic libraries at OS build time into one mega-dylib that we can pre-bind together and sign. We also use VM tricks to allow the system to rebase it on page in rather than the dynamic linker doing any work, and we pre-map it into every address space by default.
We even perform a number of additional optimizations when we build it, such as analyzing the dependencies of every executable in the base system and pre-calculating a lot of the work dyld and even the dynamic runtimes like ObjC would have to do in order to avoid doing them during app launch.
So in short, I think your ideas there have more merit than you perhaps suspect, and from experience I can say if/when you implement them you implement them you might find your views on the trade offs of dynamic vs static linking change.
Thank you.
And after reading your response, I might give up on at least one of those ideas.
And after reading your response, I might give up on at least one of those ideas.
It seems a pain point of static linking is that if a library changes you need to rebuild everyone
Maybe a stupid idea but has there ever been thought of distributing applications with source code included?
That way you could have a system that automatically rebuilds the application if a static library changes.
Meaning when a user sees “updating” it’s actually rebuilding the app underneath.
Maybe a stupid idea but has there ever been thought of distributing applications with source code included?
That way you could have a system that automatically rebuilds the application if a static library changes.
Meaning when a user sees “updating” it’s actually rebuilding the app underneath.
Correct me if I am wrong but OpenBSD introduced the constraint that syscalls must come from a specific section of memory, that assigned to their libc.
In a case like that static linking is not only unstable, as in Windows or macOS, but also impossible.
And since that has security benefits we will probably see it being migrated to other platforms in the near future.
In a case like that static linking is not only unstable, as in Windows or macOS, but also impossible.
And since that has security benefits we will probably see it being migrated to other platforms in the near future.
I care far more about about bundled
vs unbundled than dynamic vs static.
I am passionately in-favor of bundled. Unbundled distribution is what has led to software being so outrageously impossible to execute that the only sane path to distribution is to create a bundled docker image. https://xkcd.com/1987/
I am passionately in-favor of bundled. Unbundled distribution is what has led to software being so outrageously impossible to execute that the only sane path to distribution is to create a bundled docker image. https://xkcd.com/1987/
> Operating systems can statically map libraries.
I have "fond" memories of running perebaseall on cygwin that this sentence reminded me of.
I have "fond" memories of running perebaseall on cygwin that this sentence reminded me of.
I really dislike the tired re-use of the term "considered harmful". A better title may be "I don't like dynamic linking - here's why".
> However, there is one big elephant in the room: right now, there are a lot of platforms that do not fully support static linking because you have to dynamically link libc or other system interfaces. The only one I know of that supports static linking is Linux.
> The ironic thing is that operating systems only supported a stable syscall ABI, rather than requiring programs to dynamically link system libraries, they would have less problems with ABI breaks.
> With a statically-linked binary, you can copy it to another machine with a Linux that supports all of the needed syscalls, and it will just run.
I think this is a misunderstanding. With Linux there is a stable interface at the kernel boundary. Other OSes have their stable interfaces in user space. The difference is mostly irrelevant in this context. In one case you dynamically call into the kernel, in another case you dynamically link a user space library. Either way the code you're calling isn't static but the interface is.
> The ironic thing is that operating systems only supported a stable syscall ABI, rather than requiring programs to dynamically link system libraries, they would have less problems with ABI breaks.
> With a statically-linked binary, you can copy it to another machine with a Linux that supports all of the needed syscalls, and it will just run.
I think this is a misunderstanding. With Linux there is a stable interface at the kernel boundary. Other OSes have their stable interfaces in user space. The difference is mostly irrelevant in this context. In one case you dynamically call into the kernel, in another case you dynamically link a user space library. Either way the code you're calling isn't static but the interface is.
And to assume that only the syscall ABI layer is important is quite the understatement. I have software from the 90s that would work perfectly fine if it weren't for the fact that /etc/mtab is now a symlink. And that is just an example of the most smallish change -- not going to enter into the bazillions of binaries that are broken due to changes in audio APIs (even though the syscall numbers are still the same).
Static linking for future-proofing/compatibility has never been a valid argument in my experience. Not only it is much easier to fix (and debug) a broken dynamically linked binary than a statically linked one, I would even dare to say that statically linked binaries break much more frequently than dynamically linked ones when you change the rest of the system. Even when you include the breakage from libraries changing under the feet of the dynamic binaries.
Static linking for future-proofing/compatibility has never been a valid argument in my experience. Not only it is much easier to fix (and debug) a broken dynamically linked binary than a statically linked one, I would even dare to say that statically linked binaries break much more frequently than dynamically linked ones when you change the rest of the system. Even when you include the breakage from libraries changing under the feet of the dynamic binaries.
There is a distinction between a fully-static binary and a non-fully-static one, which is what I think the article refers to. A fully static binary will not need to call the dynamic linker at all, and tools such as "file" or "ldd" will identify such binaries differently. A fully static binary will also run even in the complete absence of any userspace support for its architecture (assuming the hardware and kernel do support it) - e.g. a static AArch64 binary will run on a 32-bit ARM distribution, if the CPU supports AArch64.
True, however...
> assuming the hardware and kernel do support it
Assuming kernel support is doing a lot of heavy lifting there. To put it another way, why is a syscall better than any other stable ABI? If both kernel and stable library are distributed together then why should it be considered different?
> assuming the hardware and kernel do support it
Assuming kernel support is doing a lot of heavy lifting there. To put it another way, why is a syscall better than any other stable ABI? If both kernel and stable library are distributed together then why should it be considered different?
Indeed, Linux is the outlier here. Maybe it's better. But the designers of, say, Solaris, Mac and Windows didn't think so.
Even Linux isn't really the outlier. The syscall convention is stable, but you're not statically linking the kernel into your binary.
A "library OS" like FreeRTOS does exactly that: the kernel is just another library, with functions that you call like any other static dependency. You can only run one process (the OS & your userspace code are that process).
Really the "fully static" side is only seen in practice in RTOSes and similar embedded systems work. Being able to dynamically load more than a single process is just too handy to give up entirely. I don't think the opposite extreme has even been tried (every single symbol in its own .so, even within the kernel), the overhead would be ridiculous.
A "library OS" like FreeRTOS does exactly that: the kernel is just another library, with functions that you call like any other static dependency. You can only run one process (the OS & your userspace code are that process).
Really the "fully static" side is only seen in practice in RTOSes and similar embedded systems work. Being able to dynamically load more than a single process is just too handy to give up entirely. I don't think the opposite extreme has even been tried (every single symbol in its own .so, even within the kernel), the overhead would be ridiculous.
If a kernel and library really are always distributed together, so religiously iron clan that you can and should treat them as a single object, then why are they not in fact a single object?
A syscall may not be inherently too different from any other interface, but a single interface is certainly different from two interfaces.
A syscall may not be inherently too different from any other interface, but a single interface is certainly different from two interfaces.
Author here.
Kernel support is actually easy: since Linux hardly ever removes syscalls, just build your fully-static executable on the oldest Linux you have, and then deploy it on all of your relevant machines.
The syscalls used by all of your libraries, if they work on that oldest Linux, would work on the newer ones.
In fact, this is why AppImage "Best Practices" includes building on the oldest system. [1]
[1]: https://docs.appimage.org/reference/best-practices.html?high...
Kernel support is actually easy: since Linux hardly ever removes syscalls, just build your fully-static executable on the oldest Linux you have, and then deploy it on all of your relevant machines.
The syscalls used by all of your libraries, if they work on that oldest Linux, would work on the newer ones.
In fact, this is why AppImage "Best Practices" includes building on the oldest system. [1]
[1]: https://docs.appimage.org/reference/best-practices.html?high...
If you build against, say, RHEL5, you presumably acquire vulnerabilities in relevant libraries, give up hardware support, and still can't guarantee it will run correctly. That's at least because Linux interfaces aren't stable in general, specifically the pseudo-filesystem ones, thinking of real examples.
[deleted]
Author here.
This is exactly what I was getting at. Thank you.
This is exactly what I was getting at. Thank you.
The word "static" is being conflated with "stable".
A "static interface" isn't a thing. There's stable and unstable interfaces, and static and dynamic linkage. Not strongly related.
A "static interface" isn't a thing. There's stable and unstable interfaces, and static and dynamic linkage. Not strongly related.
I'm distributing my code as a bootable USB drive.
Are there any Linux distros that actually distribute (most) software as (mostly) static binaries? Right now, the only way to do it seems to be compiling everything myself...
I think there are only experiments. From what I know Oasis Linux is furthest along https://github.com/oasislinux/oasis
Another resources is https://sta.li/ I thought there was a Archlinux-based attempt, but can't find references to that.
Another resources is https://sta.li/ I thought there was a Archlinux-based attempt, but can't find references to that.
https://sta.li/
Experimental, probably not that usable.
http://sabo.xyz/ More of a complete distro, but the dev himself says he still uses a couple of programs dynamically linked.
http://sabo.xyz/ More of a complete distro, but the dev himself says he still uses a couple of programs dynamically linked.
Why would you want that?
Simplicity. If everything is statically linked, then updating package A can never break package B.
As I see it, the entire world of Docker containers exists primarily to solve the problem that static linking already solved.
Something like Nix would work as well, and I think I would love Nix if I knew how to use it. But I don't, and I don't really have the time to learn.
As I see it, the entire world of Docker containers exists primarily to solve the problem that static linking already solved.
Something like Nix would work as well, and I think I would love Nix if I knew how to use it. But I don't, and I don't really have the time to learn.
AppImage, Flatpack, Snaps... all seem to be largely reinventing the primary qualities, good and bad, of static binaries in a new and much more complicated form.
At least a container provides more function than mere bundling, but a lot of containers actually get used merely for bundling.
At least a container provides more function than mere bundling, but a lot of containers actually get used merely for bundling.
I don't think that makes a lot of sense in a distro. If package A gets an update, it means a sufficiently bad bug was discovered in it. If it happens to be a library, you want to make sure you update every package B that uses it. Which, conveniently, the distro solves for you too.
This is true, but only if I'm actually getting everything from the distro repository and I'm able to keep all of it on the latest version. In practice, I sadly tend to run into situations where I need stuff from other places.
But then you don't want the software from the distro to be statically linked, but the software from other places. In which case it doesn't matter that much what does the distro do.
Basically, the distros aren't perfect and I sometimes run into conflicts. Also, with static binaries it would be much easier to pin specific software at old versions without affecting anything else, even just temporarily for testing.
IMO, when stuff is self-contained, everything becomes easier! This isn't to say some stuff like OpenGL can't be dynamically linked, but some day I'd love to use a distro with a static-first approach, because right now it simply isn't an option.
IMO, when stuff is self-contained, everything becomes easier! This isn't to say some stuff like OpenGL can't be dynamically linked, but some day I'd love to use a distro with a static-first approach, because right now it simply isn't an option.
I think programs in /bin used to be usually statically linked. When updating the glibc for instance, this makes the change far more doable. In general dealing with static binaries is much less pain, especially when they haven't been compiled locally/by the distributor for this particular version of the distribution.
/sbin used to be statically linked on unix. The entire point of /sbin is the independence from almost everything else so that you can rely on them even when the system boots up in a weird state.
""Static Linking Considered Harmful" Considered Harmful" Considered Harmful:
This is the blog version of Re: Re: Fwd: Re: in email and has to stop here.
See also https://meyerweb.com/eric/comment/chech.html
This is the blog version of Re: Re: Fwd: Re: in email and has to stop here.
See also https://meyerweb.com/eric/comment/chech.html
Incorrect.
Ok fine, explaination: "Re:" are not art. "Re:" are not added deliberately and for intentional artistic effect, and are not references (despite the literal) or homages.
Ok fine, explaination: "Re:" are not art. "Re:" are not added deliberately and for intentional artistic effect, and are not references (despite the literal) or homages.
Author here.
See my reply to https://meyerweb.com/eric/comment/chech.html at https://news.ycombinator.com/item?id=28737279 .
See my reply to https://meyerweb.com/eric/comment/chech.html at https://news.ycombinator.com/item?id=28737279 .
I was mostly being facetious and wrote that in jest - with a title like that you should be prepared for some OT banter ;)
This is very situational, but I have recently been part of a project that does a lot of C server-side development and we have found that static linking our non-glibc dependencies has really improved our developer experience. Using ceedling's dependency plugin[1] and producing a single "statically" linked library has made our C development much closer to using a language with a more modern package manager. Don't get me wrong, if I was trying to distribute binaries to machines I didn't control I'd definitely be willing to invest in the Linux packaging "fun", but for a server-side application it's been a good choice for our team overall.
[1] https://github.com/ThrowTheSwitch/Ceedling/tree/master/plugi...
[1] https://github.com/ThrowTheSwitch/Ceedling/tree/master/plugi...
Yes and no. In Ardour, which has a substantial dependency tree (80+ libraries), if we statically link, the edit/compile/debug cycle becomes incredibly bogged down by static linkage. It takes a long time (even with lld) to complete the link step. If we use shared libraries/dynamic linkage, the edit/compile/debug cycle is nice and fast, but application startup is a bit slower (not by as much as static linkage is slower, however).
For users, it would be better if the application was statically linked, at least in terms of startup cost. But because developers do the edit/compile/debug cycle much, much more often than users start the application, we opt for dynamic linkage.
For users, it would be better if the application was statically linked, at least in terms of startup cost. But because developers do the edit/compile/debug cycle much, much more often than users start the application, we opt for dynamic linkage.
You can dynamically link during development, and then ship statically linked binary (with LTO etc). That’s what Chrome does, for example.
It's not that simple for us (we have tried this).
I eagerly await some concrete implementations of these new ideas.
In the meantime I'm waiting for the first big vulnerability in a widely used golang or rust library to see if downstream projects pinning it also release CVEs as they should (which would probably DDoS the CVE system from the resulting avalanche) or they quietly bump it and move on (in which case their users won't get the nudge to upgrade). This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages.
And it's not that I don't completely sympathize or understand the advantages of static linking from a developer's point of view - you don't have to sell it to me there.
In the meantime I'm waiting for the first big vulnerability in a widely used golang or rust library to see if downstream projects pinning it also release CVEs as they should (which would probably DDoS the CVE system from the resulting avalanche) or they quietly bump it and move on (in which case their users won't get the nudge to upgrade). This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages.
And it's not that I don't completely sympathize or understand the advantages of static linking from a developer's point of view - you don't have to sell it to me there.
The difference between dynamically and statically linking doesn’t change the patching story so long as applications ship with all their libs and share nothing.
The only difference is that a statically linked app is a large binary blob in a directory while a dynamically linked app is multiple smaller blobs in a directory.
Whether or not statically or dynamically linked apps are the future, the idea of system wide shared libraries seems like it’s going away.
The only difference is that a statically linked app is a large binary blob in a directory while a dynamically linked app is multiple smaller blobs in a directory.
Whether or not statically or dynamically linked apps are the future, the idea of system wide shared libraries seems like it’s going away.
> The difference between dynamically and statically linking doesn’t change the patching story
I think there are some edge cases where the mainteners will always be caught between a rock and a hard place and the only way to move forward is to involve the application developers - at least as long as library compatibility can't be defined more rigidly.
As a dev, you'd like to pin the versions of your dependencies as tightly as possible, so you know the library will behave in production exactly as it did in your tests.
As a maintener, you'd like applications to specify their dependency versions as loosely as possible, so you can transparently update a dependency in case of critical patches.
This means, there can always occur a case where version x of a library has a critical vulnerability but version x+1 breaks the assumptions that the software is making.
No matter if the library is linked statically or dynamically, this situation can't be solved by the maintener alone.
Of course the GP is also correct in a sense: If applications bundle libraries or set tight version restrictions, it's now the responsibility of the developers to track critical vulnerabilities in all of their dependencies and integrate the patches as quickly as possible.
The problem with bundled libraries is that now we have to trust every single developer to get this right.
I think there are some edge cases where the mainteners will always be caught between a rock and a hard place and the only way to move forward is to involve the application developers - at least as long as library compatibility can't be defined more rigidly.
As a dev, you'd like to pin the versions of your dependencies as tightly as possible, so you know the library will behave in production exactly as it did in your tests.
As a maintener, you'd like applications to specify their dependency versions as loosely as possible, so you can transparently update a dependency in case of critical patches.
This means, there can always occur a case where version x of a library has a critical vulnerability but version x+1 breaks the assumptions that the software is making.
No matter if the library is linked statically or dynamically, this situation can't be solved by the maintener alone.
Of course the GP is also correct in a sense: If applications bundle libraries or set tight version restrictions, it's now the responsibility of the developers to track critical vulnerabilities in all of their dependencies and integrate the patches as quickly as possible.
The problem with bundled libraries is that now we have to trust every single developer to get this right.
> As a dev, you'd like to pin the versions of your dependencies as tightly as possible, so you know the library will behave in production exactly as it did in your tests.
Indeed, and this is why the shared nothing models are so attractive. Unless I was developing anything security critical such as server apps, I'd probably never want to deploy anything in classical unix style where any part of the app could be replaced behind the back of my application.
On windows and for desktop software, the risk of a failed install or the user sabotaging their own install without realizing it is more common than the risk of having an outdated version of a library. If the application is mostly or completely offline (e.g. only online for updates or documentation), then the surface area for attack is pretty small. So I'd usually go so far as validating the integrity of the app with a checksum on startup too, and refuse to start if the app isn't byte for byte exactly as expected.
This also obivously means my app has almost no risk of ending up in a package in a Linux distribution too.
Indeed, and this is why the shared nothing models are so attractive. Unless I was developing anything security critical such as server apps, I'd probably never want to deploy anything in classical unix style where any part of the app could be replaced behind the back of my application.
On windows and for desktop software, the risk of a failed install or the user sabotaging their own install without realizing it is more common than the risk of having an outdated version of a library. If the application is mostly or completely offline (e.g. only online for updates or documentation), then the surface area for attack is pretty small. So I'd usually go so far as validating the integrity of the app with a checksum on startup too, and refuse to start if the app isn't byte for byte exactly as expected.
This also obivously means my app has almost no risk of ending up in a package in a Linux distribution too.
shared ram too. it adds up.
It does, but we're already moving to a world where deployed apps are one of the very few things running in a container, which is itself running somewhere in a VM, so there's less sharing to achieve.
Yeah. In that scenario the distinction between dynamic and static linking is moot. In both cases you need to update the container image and when you do you only fixed that container image and you still need to update all the other images
There's absolutely no need for just one-ring-to-rule-them-all copy of libz.so. Shared libraries can be versioned and garbage collected like how habitat (hab) does it: in separate directories.
Exactly. The issue is not static vs dynamic; it's bundled vs unbundled.
You could probably even do an "unbundled" statically linked system, e.g. imagine Debian but everything is statically linked. Doesn't matter that everything is statically linked - if there's a vulnerability in libpng you can still easily update dependencies. Just uses more network / disk space.
You could probably even do an "unbundled" statically linked system, e.g. imagine Debian but everything is statically linked. Doesn't matter that everything is statically linked - if there's a vulnerability in libpng you can still easily update dependencies. Just uses more network / disk space.
> Doesn't matter that everything is statically linked - if there's a vulnerability in libpng you can still easily update dependencies. Just uses more network / disk space.
It's not that simple
* Someone, somewhere - probably distro volunteers - have to manage this process of rebuilding dozens, hundreds, or thousands of packages, and knowing which packages need to be updated to begin with. The amount of work required from distros (again, mostly volunteers) would be an order of magnitude greater even if the process was automated.
* Likewise, the amount of resources needed to rebuild all of those packages, and distribute them, would be orders of magnitude greater.
* The amount of extra work (both for humans and computers) required would delay the rollout of patches.
* And in the real world, a lot of users don't have limitless network bandwidth and disk space.
* And auditing systems for security vulnerabilities would be geometrically more difficult. Now you need to track a recursive bill of materials for every package on the system.
If there's a security bug in libc or something along those lines - which are fairly common- distros would need to rebuild practically every package in the OS. It's all well and good to talk about how it would be no big deal but I think if it was ever actually implemented no user would accept that trade-off.
It's not that simple
* Someone, somewhere - probably distro volunteers - have to manage this process of rebuilding dozens, hundreds, or thousands of packages, and knowing which packages need to be updated to begin with. The amount of work required from distros (again, mostly volunteers) would be an order of magnitude greater even if the process was automated.
* Likewise, the amount of resources needed to rebuild all of those packages, and distribute them, would be orders of magnitude greater.
* The amount of extra work (both for humans and computers) required would delay the rollout of patches.
* And in the real world, a lot of users don't have limitless network bandwidth and disk space.
* And auditing systems for security vulnerabilities would be geometrically more difficult. Now you need to track a recursive bill of materials for every package on the system.
If there's a security bug in libc or something along those lines - which are fairly common- distros would need to rebuild practically every package in the OS. It's all well and good to talk about how it would be no big deal but I think if it was ever actually implemented no user would accept that trade-off.
> Someone, somewhere - probably distro volunteers - have to manage this process of rebuilding dozens, hundreds, or thousands of packages, and knowing which packages need to be updated to begin with.
Packages list their dependencies. You don't have to figure it out manually; that would be insane.
Distros have automated build farms for all this stuff.
Packages list their dependencies. You don't have to figure it out manually; that would be insane.
Distros have automated build farms for all this stuff.
As I said, even with automation, there's a surprising amount of manual effort required.
I agree, but if you're bundling the dependencies why not just statically link them too? I've never understood the way macOS does it (dynamic linking, but app bundles ship with the libraries they need included).
This is also how windows apps always worked. It’s good for deployment because an update might not touch more than a few files.
Also not all language ecosystems support static linking at all, e.g .NET
Also not all language ecosystems support static linking at all, e.g .NET
> Also not all language ecosystems support static linking at all, e.g .NET
The “language ecosystem” of .NET supports static linking via .NET Native, doesn't it?
The “language ecosystem” of .NET supports static linking via .NET Native, doesn't it?
.NET native is a dead end technology that only supported old UWP apps.
There's the new single file builds but those just concatenate the dynamic libs together into one file.
There's also the new .NET linker for trimmed builds, but that requires special annotations from many libs to work.
There's the new single file builds but those just concatenate the dynamic libs together into one file.
There's also the new .NET linker for trimmed builds, but that requires special annotations from many libs to work.
Not a mac user, but one upside I can see is that this forces apps to at least declare which libraries they are using. If everything is statically linked, it might be hard to figure out where on the system a vulnerable library is being used at all.
In some cases it's required such as if you're using a closed-source library that doesn't provide a static version or if you're calling LGPL code from non-GPL-compatible code.
Are you sure about the "CVE explosion"? From the CNA counting rules https://cve.mitre.org/cve/cna/rules.html#section_7_assignmen... :
7.2.4 If multiple products are affected by the same independently fixable vulnerability, then the CNA:
a. MUST NOT assign more than one CVE ID if the products are affected, because they share the vulnerable code. The assigned CVE ID will be shared by the affected products.I'm not sure what would actually happen in reality, whether a single CVE would get endless addenda listing the packages affected by an upstream vulnerability. I certainly see plenty of new CVEs go past which are of the form "xyz had a vendored version of abc, which was vulnerable to ..."
Author here.
> I eagerly await some concrete implementations of these new ideas.
I'm working on them now! They are not public because of a licensing issue where I need to consult a lawyer first (thanks, GitHub Copilot), but they will be public as soon as that's sorted out.
> I eagerly await some concrete implementations of these new ideas.
I'm working on them now! They are not public because of a licensing issue where I need to consult a lawyer first (thanks, GitHub Copilot), but they will be public as soon as that's sorted out.
> They are not public because of a licensing issue where I need to consult a lawyer first (thanks, GitHub Copilot), but they will be public as soon as that's sorted out.
Do you mean that you built them using Copilot, or that you don't want Copilot to use them? Or something else entirely?
Do you mean that you built them using Copilot, or that you don't want Copilot to use them? Or something else entirely?
> This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages.
Maybe the problem is have a system with thousands of installed packages? If every application is in its own "space", then a security failure won't affect the rest. If you want the best security, you're going to have to do pretty big tradeoffs. If you're not ready to do them, you're going to have to live with an insecure system. I don't think there is a way around this.
Maybe the problem is have a system with thousands of installed packages? If every application is in its own "space", then a security failure won't affect the rest. If you want the best security, you're going to have to do pretty big tradeoffs. If you're not ready to do them, you're going to have to live with an insecure system. I don't think there is a way around this.
Part of the problem is not just getting all the different applications and libraries up to date, but knowing which of them need to be updated.
For a system that widely uses dynamic libs, that's fairly easy to do. Check the version of the installed library, along with any patches it might have, and you have a good idea of whether it needs to be updated.
Now, imagine there's a zlib exploit. That's a lot of things in the system that need to be updated. It's so ubiquitous that even in a system that's almost entirely built with dynamic libs, there are a few that use it as a static lib, and you'll have to make sure those are updated too (go look at the last zlib CVE and the major distro updates to it and you'll see all those packages, sometimes a couple days later as they discover them too).
A world where these are all statically compiled (or bundled separately in containers) is much more complex. It's not impossible, but you don't naturally get some of the work done for you. To some degree we're already going this way with containers though.
For a system that widely uses dynamic libs, that's fairly easy to do. Check the version of the installed library, along with any patches it might have, and you have a good idea of whether it needs to be updated.
Now, imagine there's a zlib exploit. That's a lot of things in the system that need to be updated. It's so ubiquitous that even in a system that's almost entirely built with dynamic libs, there are a few that use it as a static lib, and you'll have to make sure those are updated too (go look at the last zlib CVE and the major distro updates to it and you'll see all those packages, sometimes a couple days later as they discover them too).
A world where these are all statically compiled (or bundled separately in containers) is much more complex. It's not impossible, but you don't naturally get some of the work done for you. To some degree we're already going this way with containers though.
> Part of the problem is not just getting all the different applications and libraries up to date, but knowing which of them need to be updated.
In which was? Is it because it's hard to know which version of zlib was used in X application, or is it because there is no centralized information about this? Maybe that's an opportunity right there: build a graph of dependencies of a system, and alert when something has been compromized and which are the consequences.
In which was? Is it because it's hard to know which version of zlib was used in X application, or is it because there is no centralized information about this? Maybe that's an opportunity right there: build a graph of dependencies of a system, and alert when something has been compromized and which are the consequences.
> build a graph of dependencies of a system
You mean in the sense that every application that comes with statically linked libraries should also come with some metadata indicating which libraries/versions are in there?
This would work, but then you have the problem metadata always has: It can get out of sync with the actual thing, developers can forget to properly maintain it (or even to provide it at all) etc etc.
You mean in the sense that every application that comes with statically linked libraries should also come with some metadata indicating which libraries/versions are in there?
This would work, but then you have the problem metadata always has: It can get out of sync with the actual thing, developers can forget to properly maintain it (or even to provide it at all) etc etc.
You're right, I didn't answer the question of "who should maintain that stuff?". I was imagining something like libraries in JS or Rust, where people pin the dependencies of their libraries, so with this, you can build a graph of dependencies. I guess I have a very "github" centric vision of things and that most software isn't built this way, which would complicate things.
To some degree most distro's have this graph and it's somewhat queryable (rpm has to agilities for this, if not easily consumed). The difference is in how accurate and queryable it is, and how easy it is to be fairly sure everything is up to date.
It's not a show stopping problem, but it is something that should be considered and thought closely about. This is a security adjacent problem, and changes in it can have consequences that are larger than "I'm running a library that doesn't have the feature I want".
It's not a show stopping problem, but it is something that should be considered and thought closely about. This is a security adjacent problem, and changes in it can have consequences that are larger than "I'm running a library that doesn't have the feature I want".
I believe when heartbleed happened quite a bunch of go progams were yoinking in openssl because the go ssl tooling was still relatively young.
I have no idea what actually happened there, I just recall some moderately annoyed sysadmins figuring out how to rebuild go applications by developers who'd since left the company.
(this is not a shot at go, this is merely a "there might already be -some- data out there" wrt your question)
I have no idea what actually happened there, I just recall some moderately annoyed sysadmins figuring out how to rebuild go applications by developers who'd since left the company.
(this is not a shot at go, this is merely a "there might already be -some- data out there" wrt your question)
The design of the Go build and packages system kinda renders that point moot because the expectation is that provided you follow Go's guidelines then a Go application should always be easily recompilable, and all external dependencies should always be distributed in source-form, not as binaries (e.g. https://github.com/golang/go/issues/2775 ).
> provided you follow Go's guidelines then a Go application should always be easily recompilable, and all external dependencies should always be distributed in source-form, not as binaries
So it's a human solution, and not a technical one, which means that it can and will fail.
So it's a human solution, and not a technical one, which means that it can and will fail.
That's like saying condoms are ineffective because you included people too lazy to put them on in the first place in your calculations.
Provided you (and your org) use Go as Google intends then that isn't an issue.
Provided you (and your org) use Go as Google intends then that isn't an issue.
> That's like saying condoms are ineffective because you included people too lazy to put them on in the first place in your calculations.
I think it's closer to say that the existance of condoms won't make AIDS and unwanted pregnancies disappear, even if they were 100% effective. Which, if you're trying to eliminate completly AIDS, is a fair claim to make.
My point is that you shouldn't assume that everyone is going to follow the Go guidelines, because some people won't, and you don't want your security to rely on that.
I think it's closer to say that the existance of condoms won't make AIDS and unwanted pregnancies disappear, even if they were 100% effective. Which, if you're trying to eliminate completly AIDS, is a fair claim to make.
My point is that you shouldn't assume that everyone is going to follow the Go guidelines, because some people won't, and you don't want your security to rely on that.
Right, but if every application did distribute as source and could just be recompiled, there would be fewer complaints about the difficulties distributing Linux binaries
> This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages.
Is updating a library with thousands of dependents, without individually testing those dependents, any more feasible?
Is updating a library with thousands of dependents, without individually testing those dependents, any more feasible?
That's the choice you're making everytime you add another dependency--you are now bound to that repo and all of it's upstreams. You need to follow along send stay on top of Caves, API deprecations, etc. I don't think people weigh this cost properly when doing a build vs integrate decision. Turns out dependency management is a lot like the "when you sleep with someone you're sleeping with everyone they've ever slept with" ideology.
Well, it's what traditional distributions do every day.
In NixOS, we try to work the test suites into the build process and rebuilds happen when any dependencies change, so I'd like to think we get (almost) the best of both worlds.
In NixOS, we try to work the test suites into the build process and rebuilds happen when any dependencies change, so I'd like to think we get (almost) the best of both worlds.
> Well, it's what traditional distributions do every day.
Which coincidentally is why I'm not using Linux right now. :) A move towards static linking in the Linux world would go a long way towards changing that.
(Everything I've read about NixOS seems great, thanks for helping with that—it's just that it also has a large barrier to entry.)
Which coincidentally is why I'm not using Linux right now. :) A move towards static linking in the Linux world would go a long way towards changing that.
(Everything I've read about NixOS seems great, thanks for helping with that—it's just that it also has a large barrier to entry.)
(I'm merely a lowly maintainer in NixOS, shoulders of giants etc.)
You would have to do that anyway. With static linking, you also have to rebuild those dependents and make sure their users update them as well.
I would rather the developers rebuild the dependents, test them, make any necessary fixes, and send me the updated (presumably static) binaries. As opposed to me, as an individual user with no familiarity with the code base, just switching out the pieces and preying nothing goes wrong.
As a small developer who handles my own support, I agree with this sentiment completely!!
You'll have the opposite problem though, that you have to trust every single developer to properly manage their dependencies and publish timely patches if there is a vulnerability.
> As opposed to me, as an individual user with no familiarity with the code base, just switching out the pieces and preying nothing goes wrong.
Wouldn't this be the task of your distro's mainteners, not you? (Who will have familiarity with all codebases as well as options to contact the original developers if there is an unsolvable issue)
> As opposed to me, as an individual user with no familiarity with the code base, just switching out the pieces and preying nothing goes wrong.
Wouldn't this be the task of your distro's mainteners, not you? (Who will have familiarity with all codebases as well as options to contact the original developers if there is an unsolvable issue)
> Wouldn't this be the task of your distro's mainteners, not you?
Sure, and when they’re done, they can recompile the static binary and send me an updated copy.
I don’t want to update until they’ve finished testing the dependents. At least with static binaries, I can update some of the dependents sooner as the maintainers work through their lists.
Sure, and when they’re done, they can recompile the static binary and send me an updated copy.
I don’t want to update until they’ve finished testing the dependents. At least with static binaries, I can update some of the dependents sooner as the maintainers work through their lists.
And if the upstream developers are unresponsive, or only provide the bump with the latest bleeding-edge version which you can't upgrade to yet?
On a system where security is critical, I would probably need to re-evaluate my use of that software.
It doesn't necessarily need to be the upstream developer, it could be some other organization analogous to a distro maintainer. What matters is that someone who is familiar with the software and code base has actually tested the update in a purposeful way.
It doesn't necessarily need to be the upstream developer, it could be some other organization analogous to a distro maintainer. What matters is that someone who is familiar with the software and code base has actually tested the update in a purposeful way.
It would certainly make the embedded industry more secure if it was always running the latest version of everything. Or at least the ISPs, because imagine the bandwidth they're be able to charge for as everything from your washing machine to your watch to your lightbulbs to your car downloads new images hourly.
Just imagining the joy of broken firmware updates being pushed out to all my lightbulbs...
Unfortunately not even Linux allows full static linking (as far as I'm aware at least) as soon as libraries like OpenGL are involved.
Dynamic linking with core system libraries which are guaranteed to exist(!) is fine though, for everything else dynamic linking mostly has downsides.
Also: it's only Linux where this is an issue, because it's nearly impossible to build small self-contained executables which run on another Linux machine, thanks to glibc's versioning mess (at least there's MUSL as an alternative to create portable command line tools).
Dynamic linking with core system libraries which are guaranteed to exist(!) is fine though, for everything else dynamic linking mostly has downsides.
Also: it's only Linux where this is an issue, because it's nearly impossible to build small self-contained executables which run on another Linux machine, thanks to glibc's versioning mess (at least there's MUSL as an alternative to create portable command line tools).
At least one can statically link the application except for forcibly-dynamic libs like libGL (also, apparently using dlopen() is often faster than naive dynamic linking...)
Well yeah, the GL calls are actually different on different machines right? Isn't that the absolute first consideration when deciding if you should dynamically link?
That's normally what kernel abstractions are for, but unfortunately for various historical reasons graphics API often bypass the kernel in part or in full. There's also the problem that graphics are often very performance-sensitive so costly abstractions may not be desirable.
I'd rather expect that the 'user-facing' GL implementation (GL2.x, GL3.x, ...) is identical and can be statically linked, and that only the graphics driver layer differs between machines (which would be similar to an executable which links the C library statically, which in turn talks to the operating system through syscalls).
But to be fair, dynamically linked GL wouldn't be a problem, if this wouldn't also pull in a dynamically linked glibc (it always comes back to glibc unfortunately).
But to be fair, dynamically linked GL wouldn't be a problem, if this wouldn't also pull in a dynamically linked glibc (it always comes back to glibc unfortunately).
> I'd rather expect that the 'user-facing' GL implementation (GL2.x, GL3.x, ...) is identical and can be statically linked, and that only the graphics driver layer differs between machines (which would be similar to an executable which links the C library statically, which in turn talks to the operating system through syscalls).
The problem is that the "graphics driver layer" (except a small amount of security-sensitive pieces) runs in user space, in the same process which is doing the OpenGL/Vulkan/Metal/etc calls. The way modern graphics hardware works is that the application writes a very complex and hardware-dependent set of command buffers, which are then submitted to the hardware, and only this final submission goes through the kernel (for security reasons). If the application had to call the kernel for each and every step of the creation of the command buffers, it would kill the performance.
The problem is that the "graphics driver layer" (except a small amount of security-sensitive pieces) runs in user space, in the same process which is doing the OpenGL/Vulkan/Metal/etc calls. The way modern graphics hardware works is that the application writes a very complex and hardware-dependent set of command buffers, which are then submitted to the hardware, and only this final submission goes through the kernel (for security reasons). If the application had to call the kernel for each and every step of the creation of the command buffers, it would kill the performance.
Isn't this the point of libglvnd?
Didn't Go have to throw in the towel on not dynamically linking libc on macOS as there's no guarantee of syscall stability at all?
I don't know about macOS, but they also did this on OpenBSD. The reason is a recent security addition in OpenBSD, where they made it illegal (as in: violate it, and the process will be killed) to issue syscalls to the kernel from anything other than libc.
https://lwn.net/Articles/806776/ discusses that feature and says “Switching Go to use the libc wrappers (as is already done on Solaris and macOS)”, so yes, they did that on macOS, too.
Reading https://golang.org/doc/go1.11#runtime, that’s fairly recent (since go 1.11)
Reading https://golang.org/doc/go1.11#runtime, that’s fairly recent (since go 1.11)
Don't know, I only know that I can build a macOS command line tool on one machine, and run it on a different machine without problems, while on Linux it's very easy to run into the glibc version compatibility problem.
I always understood the glibc versioning problem to be solved by just building on the oldest common ancestor. If you have rhel5 through 8, you build on rhel5 and move on. If you depend on more than glibc... Well rhel5 is less fun.
Author here.
Unfortunately, building on the oldest common ancestor will still not save you from ABI breaks. An example is [1]. If the size of `intmax_t` changes between that oldest ancestor and the machines you are running on, get ready for pain.
[1]: https://thephd.dev/intmax_t-hell-c++-c
Unfortunately, building on the oldest common ancestor will still not save you from ABI breaks. An example is [1]. If the size of `intmax_t` changes between that oldest ancestor and the machines you are running on, get ready for pain.
[1]: https://thephd.dev/intmax_t-hell-c++-c
Indeed, the only difference is that you might be using less optimal system calls, but there is no reason to be blocked and not target ancient kernels. I linked to 2.6 this year and everything ran fine.
It's also tricky if you need to call gethostbyname() and expect system local behavior.
[deleted]
[deleted]
I wish I had three hours to waste on why almost all of these points are either innacurate or willfully misleading. But I don't, so I'll just point out the single biggest reason why static linking is harmful.
Encryption libraries.
If OpenSSL, or LibreSSL, or Go's encryption modules, or any gigantic encryption library has a vulnerability, you basically have to recompile, distribute, and then have your users download, every single god damn networking program.
But it's worse than that. Because everyone wants to ship a statically compiled Go app, nobody packages for the distros. So now every user needs to go track down where they downloaded each of their static Go apps, safely download the update, verify its checksum or signature via a 3rd party, and upgrade the app.
That is a logistical nightmare. Not only are apps going to remain vulnerable for way, way longer, more users will be compromised by phishing and other attacks because there's no package manager to safely automate all these updates (distros sign and verify packages for you).
I've actually written a Shell quasi-package manager/installer just for statically compiled apps (cliv) so in theory that problem could be somewhat solved... But nobody even knows about that program, so the manual update problem still stands.
Encryption libraries.
If OpenSSL, or LibreSSL, or Go's encryption modules, or any gigantic encryption library has a vulnerability, you basically have to recompile, distribute, and then have your users download, every single god damn networking program.
But it's worse than that. Because everyone wants to ship a statically compiled Go app, nobody packages for the distros. So now every user needs to go track down where they downloaded each of their static Go apps, safely download the update, verify its checksum or signature via a 3rd party, and upgrade the app.
That is a logistical nightmare. Not only are apps going to remain vulnerable for way, way longer, more users will be compromised by phishing and other attacks because there's no package manager to safely automate all these updates (distros sign and verify packages for you).
I've actually written a Shell quasi-package manager/installer just for statically compiled apps (cliv) so in theory that problem could be somewhat solved... But nobody even knows about that program, so the manual update problem still stands.
> If OpenSSL, or LibreSSL, or Go's encryption modules, or any gigantic encryption library has a vulnerability, you basically have to recompile, distribute, and then have your users download, every single god damn networking program.
Good, then you'd know that these 50 of your networking apps have each been tested by their developers' test suites and have no bugs that have been introduced as a part of this dependency. Essentially, there would be a server farm out there that'd run all of the tests so you wouldn't have to deal with prod breaking because of some new dependency version not taking every possible configuration into account.
Of course, that implies:
> But it's worse than that. Because everyone wants to ship a statically compiled Go app, nobody packages for the distros.
In my eyes, that just adds to the above - there are so many deployment targets out there, all of these different packaging systems that are used by different distros and all have numerous idiosyncrasies as opposed to a single good package manager, that people have been fed up with it and now just either ship Docker containers or expect you to download binaries.
Good, then you'd know that these 50 of your networking apps have each been tested by their developers' test suites and have no bugs that have been introduced as a part of this dependency. Essentially, there would be a server farm out there that'd run all of the tests so you wouldn't have to deal with prod breaking because of some new dependency version not taking every possible configuration into account.
Of course, that implies:
- that there are automated builds in place
- that there are automated tests in place
- that both of the above are run regularly, on every dependency update
- that both of the above are actually meaningful (think close to 100% functionality coverage in tests)
> But it's worse than that. Because everyone wants to ship a statically compiled Go app, nobody packages for the distros.
In my eyes, that just adds to the above - there are so many deployment targets out there, all of these different packaging systems that are used by different distros and all have numerous idiosyncrasies as opposed to a single good package manager, that people have been fed up with it and now just either ship Docker containers or expect you to download binaries.
If you ever statically link OpenSSL, your binary now has a builtin expiration date. You are doing your users a disservice.
To be fair, if you dynamically link to OpenSSL, your binary has a builtin expiration date too, and it's very likely the same date.
OpenSSL ABI has been fairly stable within a given series.
Something compiled against 1.1.0a (Sep 2016) has a fair chance of running on a system that has 1.1.0l (Aug 2021). 5 years of binary compatibility is nothing to be sniffed at for a C library.
https://abi-laboratory.pro/index.php?view=timeline&l=openssl
Something compiled against 1.1.0a (Sep 2016) has a fair chance of running on a system that has 1.1.0l (Aug 2021). 5 years of binary compatibility is nothing to be sniffed at for a C library.
https://abi-laboratory.pro/index.php?view=timeline&l=openssl
Funny because I've seen incompatible changes twice on this timeframe.
The last one I remember. It was moving the API that did version negotiation from recommended to deprecated and then removed status. But yeah, the API itself didn't change. All that changed is the recommended way to interact with the library, and the old one removed.
The last one I remember. It was moving the API that did version negotiation from recommended to deprecated and then removed status. But yeah, the API itself didn't change. All that changed is the recommended way to interact with the library, and the old one removed.
The last release of IMAPSize e.g. dates from 2009 and can no longer connect to some IMAP servers that presumably don't support older SSL/TLS-versions, but by replacing the OpenSSL-DLLs with a recentish versions, everything works fine again.
Not really. I have made updated versions of libSSL that allow use of new ciphers with the old ABI, for example. (Retro purposes). But for example this has been done to extend the life of other binary-mostly platforms (e.g. webOS). As long as I can change libSSL separately, this is trivial.
If you just update an encryption library and ship only that… then who’s tested all the applications against the new version?
Not the library author's problem. That's the consumer's problem.
I'm not saying it's great, but it's how it works.
I'm not saying it's great, but it's how it works.
> That's the consumer's problem.
Well this is the explanation for why people are static linking! They don’t want to run someone else’s entirely untested code and bear all responsibility for any problems.
Well this is the explanation for why people are static linking! They don’t want to run someone else’s entirely untested code and bear all responsibility for any problems.
For the record? I'm totally fine with that. As I'm basically one of the static linking group, or "use dynamic libraries in a way non-distinguishable from static linking".
If I'm relying on it, it's getting tested and I'm not going to dork with it without good reason once it is.
If I'm relying on it, it's getting tested and I'm not going to dork with it without good reason once it is.
Author here.
I think you are strawmaning my arguments.
Personally, I would include encryption libraries in the "system libraries" (I used that term on purpose) that could be dynamically linked.
However, even if they are, you still want to recompile every user whenever that library is updated because of possible ABI and API breaks. And some of those encryption libraries have a history of API breaks.
Also, I am building the ideas I laid out in the post, which should make it easier to handle dynamically-linked and statically-linked encryption libraries.
I think you are strawmaning my arguments.
Personally, I would include encryption libraries in the "system libraries" (I used that term on purpose) that could be dynamically linked.
However, even if they are, you still want to recompile every user whenever that library is updated because of possible ABI and API breaks. And some of those encryption libraries have a history of API breaks.
Also, I am building the ideas I laid out in the post, which should make it easier to handle dynamically-linked and statically-linked encryption libraries.
Personal end user perspective: troubleshooting dylib errors is the most common cause of wanting to throw the computer out the window and quit programming.
"end user perspective" + "quit programming"? Do you troubleshoot dylib errors on your regular end user software and just happen to be a programmer unrelated to that?
(If you're troubleshooting dylib errors on things you're developing, that's not an end user perspective?)
(If you're troubleshooting dylib errors on things you're developing, that's not an end user perspective?)
There’s a middle level of “troubleshooting dylib errors on open source software I’m installing for personal use because I’m a masochist who isn’t happy with closed source software”
Not sure that's a solvable problem, or even a problem that should be solved. It's the distributions' job to create packages to build a working whole system. If you decide to take that on yourself… I mean… yeah, you gotta learn how to handle DSO issues. Because you decided you want to do it yourself.
I don't think I've ever run into DSO problems with any reasonable distro packaging (which, in this case, excludes old Gentoo — current Gentoo doesn't have those problems anymore either.)
I don't think I've ever run into DSO problems with any reasonable distro packaging (which, in this case, excludes old Gentoo — current Gentoo doesn't have those problems anymore either.)
But if the distro doesn't package software you want to use, either because it only has an out of date version, modified version, or was just never packaged, then you're left dealing with this nonsense.
In saner OS's where they don't rely on unpaid third party middlemen between the developer and the user this is rarely a problem.
In saner OS's where they don't rely on unpaid third party middlemen between the developer and the user this is rarely a problem.
There’s nothing especially sane about combining and testing code for the first time ever from different maintainers who weren’t even talking to each other. I’m glad that we don’t have to hire anyone full-time to do that from scratch just for us.
> In saner OS's where they don't rely on unpaid third party middlemen between the developer and the user this is rarely a problem.
Windows seems to fit your criteria for an OS without unpaid third parties, and there still exists a thing known as DLL Hell.
Windows seems to fit your criteria for an OS without unpaid third parties, and there still exists a thing known as DLL Hell.
I haven't really encountered DLL hell in Windows for the last 15 years or so. Most apps simply bundle all their dependencies locally and don't install them system wide.
It's true, DLL Hell is mostly a thing of a past. The fact the Windows apps "fix" it by maintaining their own dependencies completely isolated from system-level libraries isn't exactly what I'd call a great solution. On the flip side, an application running in isolation from the system in a container including all the necessary dependencies is pretty much Docker, so maybe it's not terrible?
> The fact the Windows apps "fix" it by maintaining their own dependencies completely isolated from system-level libraries isn't exactly what I'd call a great solution.
Sadly ran into an issue with this today. A program I wanted to use I guess forgot to ship the dependencies in the provided binaries of most recent version. I could maybe track them down, but many of not all of them would have to be built from source, which I’d rather not bother with.
Sadly ran into an issue with this today. A program I wanted to use I guess forgot to ship the dependencies in the provided binaries of most recent version. I could maybe track them down, but many of not all of them would have to be built from source, which I’d rather not bother with.
Yes. If you've never had a clashing Debian package build before, you've not been paying attention. I've made it my mission in life to try to get people more aware of linkers, and the fact that when you build something the one way it works on your system it doesn't mean it will work that way on someone else's.
I've never had a clashing Debian package build before, and I build quite a few Debian packages. Some for a wider audience and some for my own use.
All packages that leave the confines of my own systems are built the "correct" way, in a clean minimal chroot with sbuild/schroot/cowbuilder. The docs on how to set that up are pretty good, and after you've done it once it's pretty easy to repeat for extra target distros (e.g. older Debian, Ubuntu.)
What would you point out to me as part of your mission?
[Just to be clear, a package is a package for a particular target distro. There's no such thing as "a .deb for software XYZ". Only "a buster .deb for software XYZ", or "a bullseye .deb for software XYZ", etc.]
All packages that leave the confines of my own systems are built the "correct" way, in a clean minimal chroot with sbuild/schroot/cowbuilder. The docs on how to set that up are pretty good, and after you've done it once it's pretty easy to repeat for extra target distros (e.g. older Debian, Ubuntu.)
What would you point out to me as part of your mission?
[Just to be clear, a package is a package for a particular target distro. There's no such thing as "a .deb for software XYZ". Only "a buster .deb for software XYZ", or "a bullseye .deb for software XYZ", etc.]
End user of a linker?
> End user of a linker?
I'd argue that if you're not packaging your software or testing your software's dependencies, either you're doing something extremely exotic that lies far outside anyone's happy path or "dylib error" should not even be a keyword in your vocabulary.
I'd argue that if you're not packaging your software or testing your software's dependencies, either you're doing something extremely exotic that lies far outside anyone's happy path or "dylib error" should not even be a keyword in your vocabulary.
Roughly the same thing happens on Windows and is called "DLL hell", occasionally witnessed by end users.
DLL Hell ceased to be a practical concern over a decade ago, particularly given that Windows provides tight control over its dynamic linking search order.
https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-...
DLL Hell is not a linking problem, it's a packaging problem.
https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-...
DLL Hell is not a linking problem, it's a packaging problem.
See also: DLL Hell in Windows and, more generally Dependency Hell. Dynamic linking errors are a subset of general dependency problems.
As others have already mentioned elsewhere, DLL hell hasn't really been a thing anymore on Windows for years. I personally can remember it, but that's more than 15 years ago.
It's true, but only because every app essentially isolates itself from the system and carries around its own dependency bundle. I guess that's kind of what Docker containers are, so it's hard to be critical.
Why are we still developing software like we used to 40 years ago?
Why not just import external dependencies into your project at a function/class level rather than at a package one: if you only use FooPackage.BarClass and FooPackage.bazMethod(), you should be able to choose to make your project depend on just those two things, ideally in a completely transparent way by your IDE.
Then having to manage the full scope of packages and their versions becomes less relevant, because the IDE can check whether a new version has been released in the package manager of choice, check whether those two things have changed and if they have, then demand that the developer have a look at the code changes to refactor code if necessary, otherwise not requiring any action on their part. Furthermore, if you ever need to migrate to a different package or even rewrite the functionality itself, you could just look at the few signatures that you use, rather than having to remove the package entirely and see what breaks in your IDE. Why can't we have itemized lists of everything that's called and everything that we depend on, as opposed to the abstraction of entire packages?
Better yet, why even depend on binary blobs or hidden code (that's ignored by your IDE) in the first place? Why not just download the source for every package that you use and upon updates be able to review the code changes to the package much like a regular Git diff?
Of course, this would probably require getting rid of reflection and other forms of dynamic code, which i fully support, since those have never been good for much in the first place and only destroy any hopes of determinism and full control flow analysis.
As for the possible counterargument of this being hard to do: it wouldn't really be with more granular packages. Instead of trying to shove an ecosystem inside of a single package, why not split it into 10-20 bits of reusable code instead? Smaller packages, which would be easier to review and manage.
Context: i dislike how Spring and Spring Boot in Java force a huge ecosystem of fragile dependencies upon you, with their reflection which ensures that your apps will break at runtime, for example, when moving from Spring Boot 1.5 to 2.0. Furthermore, in the JS world, the node_modules folders are probably 10-1000x too large for what's actually necessary to display a webpage with some interactive behaviour.
Disclaimer: i have no delusions about any of the above being technically feasible right now. Perhaps "Why?" would be a good question to ask. In my eyes, perhaps the industry is too set in its current ways and as long as we don't have languages and entire ecosystems that approach established practices from a wildly different angle, no one can actually contest the decades of already built packages and tools.
On the bright side, WireGuard kind of displaced OpenVPN somewhat and there are numerous benefits to it being smaller, which is a good example of getting rid of bloated software. Furthermore, Nix may or may not do that to alternative approaches to package management, but its usage still remains really low.
In my eyes, the only long term solution is to be able to tell exactly what's necessary for your code to build and work, and to test every dependency update that comes out against this automated process. Static dependencies but at the speed of updates of dynamic dependencies. Then you'd just have to wait for a day until the devs fix the newest regressions of a new dependency release before getting a new statically linked app, as opposed to using dynamic linking and finding that some dependency breaks your app as it happens in prod.
Why not just import external dependencies into your project at a function/class level rather than at a package one: if you only use FooPackage.BarClass and FooPackage.bazMethod(), you should be able to choose to make your project depend on just those two things, ideally in a completely transparent way by your IDE.
Then having to manage the full scope of packages and their versions becomes less relevant, because the IDE can check whether a new version has been released in the package manager of choice, check whether those two things have changed and if they have, then demand that the developer have a look at the code changes to refactor code if necessary, otherwise not requiring any action on their part. Furthermore, if you ever need to migrate to a different package or even rewrite the functionality itself, you could just look at the few signatures that you use, rather than having to remove the package entirely and see what breaks in your IDE. Why can't we have itemized lists of everything that's called and everything that we depend on, as opposed to the abstraction of entire packages?
Better yet, why even depend on binary blobs or hidden code (that's ignored by your IDE) in the first place? Why not just download the source for every package that you use and upon updates be able to review the code changes to the package much like a regular Git diff?
Of course, this would probably require getting rid of reflection and other forms of dynamic code, which i fully support, since those have never been good for much in the first place and only destroy any hopes of determinism and full control flow analysis.
As for the possible counterargument of this being hard to do: it wouldn't really be with more granular packages. Instead of trying to shove an ecosystem inside of a single package, why not split it into 10-20 bits of reusable code instead? Smaller packages, which would be easier to review and manage.
Context: i dislike how Spring and Spring Boot in Java force a huge ecosystem of fragile dependencies upon you, with their reflection which ensures that your apps will break at runtime, for example, when moving from Spring Boot 1.5 to 2.0. Furthermore, in the JS world, the node_modules folders are probably 10-1000x too large for what's actually necessary to display a webpage with some interactive behaviour.
Disclaimer: i have no delusions about any of the above being technically feasible right now. Perhaps "Why?" would be a good question to ask. In my eyes, perhaps the industry is too set in its current ways and as long as we don't have languages and entire ecosystems that approach established practices from a wildly different angle, no one can actually contest the decades of already built packages and tools.
On the bright side, WireGuard kind of displaced OpenVPN somewhat and there are numerous benefits to it being smaller, which is a good example of getting rid of bloated software. Furthermore, Nix may or may not do that to alternative approaches to package management, but its usage still remains really low.
In my eyes, the only long term solution is to be able to tell exactly what's necessary for your code to build and work, and to test every dependency update that comes out against this automated process. Static dependencies but at the speed of updates of dynamic dependencies. Then you'd just have to wait for a day until the devs fix the newest regressions of a new dependency release before getting a new statically linked app, as opposed to using dynamic linking and finding that some dependency breaks your app as it happens in prod.
> if you only use FooPackage.BarClass and FooPackage.bazMethod()...
That's how it works already with static linking (just under the hood), the linker will discard unused code and data (with LTO this happens automatically for all dead code and data, otherwise some special compiler/linker/librarian-options may be needed).
A DLL on the other hand cannot predict what API functions will actually be called on it, so everything must be included.
That's how it works already with static linking (just under the hood), the linker will discard unused code and data (with LTO this happens automatically for all dead code and data, otherwise some special compiler/linker/librarian-options may be needed).
A DLL on the other hand cannot predict what API functions will actually be called on it, so everything must be included.
> Why not just import external dependencies into your project at a function/class level rather than at a package one: if you only use FooPackage.BarClass and FooPackage.bazMethod(), you should be able to choose to make your project depend on just those two things, ideally in a completely transparent way by your IDE.
TFA is about programs that require a linker at either compile and/or run time. The references in the rest of your comment to Java and JS suggest that your experience with programming in languages that require linking against compiled libraries might be limited.
What you describe is precisely what the linker is for! It looks at the code you wrote (and compiled), sees that it calls FooPackage.bazMethod(), and then goes and looks for that call along the linker search path using a combination of implicit and explicit library names. If it's the static linker, it imports the code for that call, and then recurses to check the dependencies of FooPackage.bazMethod(). If it's the dynamic linker, it does the same, except differently (runtime pointer swizzling etc. etc)
When you link your program against libfootastic.a, the linker is doing exactly what you suggest, and in most IDEs that step is going to happen automatically. When you link your program against libfootastic.so, the linker will do exactly what you suggest, but at runtime.
> Better yet, why even depend on binary blobs or hidden code (that's ignored by your IDE) in the first place? Why not just download the source for every package that you use and upon updates be able to review the code changes to the package much like a regular Git diff?
Many large end-user desktop applications do precisely this. For Ardour (I'm the lead dev), our build system downloads 80+ source code packages and builds them locally to create a 2GB installed dependency tree of compiled libraries and ancillary files. There are a lot of arguments against doing this, and some arguments in favor.
However, we can't compile each file from each dependency as part of the Ardour build because the build systems for those dependencies are complex and independent. There's no way to force the developers of libfftw (Fastest Fourier Transform in the West) to somehow prepare their code structures so that we can trivially replicate their entire build system as part of ours. If there was only a single build system for EVERYTHING, sure, maybe this would be feasible. But there isn't, and I'm completely certain that there never will be.
TFA is about programs that require a linker at either compile and/or run time. The references in the rest of your comment to Java and JS suggest that your experience with programming in languages that require linking against compiled libraries might be limited.
What you describe is precisely what the linker is for! It looks at the code you wrote (and compiled), sees that it calls FooPackage.bazMethod(), and then goes and looks for that call along the linker search path using a combination of implicit and explicit library names. If it's the static linker, it imports the code for that call, and then recurses to check the dependencies of FooPackage.bazMethod(). If it's the dynamic linker, it does the same, except differently (runtime pointer swizzling etc. etc)
When you link your program against libfootastic.a, the linker is doing exactly what you suggest, and in most IDEs that step is going to happen automatically. When you link your program against libfootastic.so, the linker will do exactly what you suggest, but at runtime.
> Better yet, why even depend on binary blobs or hidden code (that's ignored by your IDE) in the first place? Why not just download the source for every package that you use and upon updates be able to review the code changes to the package much like a regular Git diff?
Many large end-user desktop applications do precisely this. For Ardour (I'm the lead dev), our build system downloads 80+ source code packages and builds them locally to create a 2GB installed dependency tree of compiled libraries and ancillary files. There are a lot of arguments against doing this, and some arguments in favor.
However, we can't compile each file from each dependency as part of the Ardour build because the build systems for those dependencies are complex and independent. There's no way to force the developers of libfftw (Fastest Fourier Transform in the West) to somehow prepare their code structures so that we can trivially replicate their entire build system as part of ours. If there was only a single build system for EVERYTHING, sure, maybe this would be feasible. But there isn't, and I'm completely certain that there never will be.
Yep wrangling the dependencies really becomes a tremendous pain at scale. Chromium for example does the same thing and they have enough engineering backing to be able to throw enough manpower into it so that that can get each and every single dependency in the same build set using the same build files and tools (gyp/gn/ninja). While feasible for an org such as Google this is totally unfeasible for a small shop with Limited engineering man hours. Really wish there was an easier way to deal with all of this. Sigh
As an end user, I would LOVE to be able to run a calculator app without having to pull in 500M of kde libraries and even daemons.
But this is pretty much what a static binary already does, did, 40 years ago. The developer has access to the full universe, but the only thing that ends up in the binary are the bits they actually used. Problem solved, 40 years ago.
But this is pretty much what a static binary already does, did, 40 years ago. The developer has access to the full universe, but the only thing that ends up in the binary are the bits they actually used. Problem solved, 40 years ago.
Joe Armstrong had some similar thoughts: "Why do we need modules at all?".
https://erlang.org/pipermail/erlang-questions/2011-May/05876...
https://erlang.org/pipermail/erlang-questions/2011-May/05876...
You know, the "all functions have unique distinct names" idea is pretty interesting!
Honestly, that reminds me of the PHP standard library a bit, which feels oriented towards procedural programming at times. Now, the language and the actual implementation of the functions aside, i found that pattern really pleasant to work with, especially since it let me work more in line with functional programming principles as well - pure functions and no global state (even if passing data around was more difficult).
Now, i won't say that the approach fits every project or domain out there, but at the very least having a clear idea of what your code depends on and what's going on inside of your code base is an idea that i still stand by. Sadly, the only tool that i found that at least tries to make this easier was Sourcetrail, which is essentially dead now: https://www.sourcetrail.com/
Honestly, that reminds me of the PHP standard library a bit, which feels oriented towards procedural programming at times. Now, the language and the actual implementation of the functions aside, i found that pattern really pleasant to work with, especially since it let me work more in line with functional programming principles as well - pure functions and no global state (even if passing data around was more difficult).
Now, i won't say that the approach fits every project or domain out there, but at the very least having a clear idea of what your code depends on and what's going on inside of your code base is an idea that i still stand by. Sadly, the only tool that i found that at least tries to make this easier was Sourcetrail, which is essentially dead now: https://www.sourcetrail.com/
ldd is not the dynamic linker, it's only a tool to debug the process of dynamic linking. The dynamic linker is ld-linux.so (exact name depends on glibc version, architecture, etc.)
Also, I think the linked article [1] about the security of ldd is somewhat useless. The ldd(1) manpage [2] is very explicit about the security of ldd and tells you not to run ldd on untrusted executables:
> [...] Thus, you should never employ ldd on an untrusted executable, since this may result in the execution of arbitrary code.
It's a little amusing how the linked blog post explains how to create a malicious executable that runs arbitrary code when inspected with ldd, noting that "I researched this subject thoroughly and found that it's almost completely undocumented. I have no idea how this could have gone unnoticed for such a long time." and concluding with "Never run ldd on unknown executables!" - all while the manpage literally mentions that precise known limitation in the third paragraph!
To be fair, you could argue that this limitation is not widely known and people should be made aware of the risks of ldd, but on the other hand you can hardly criticize ldd when its manpage is that explicit about the issue.
[1] https://catonmat.net/ldd-arbitrary-code-execution
[2] https://www.man7.org/linux/man-pages/man1/ldd.1.html