To secure the supply chain, you must properly fund it(ariadne.space)
ariadne.space
To secure the supply chain, you must properly fund it
https://ariadne.space/2021/12/11/to-secure-the-supply-chain-you-must-properly-fund-it/
58 comments
> Commercial software with well-paid programmer employees also have long lists of CVE/RCE including MS Windows, Azure, AWS, Adobe PDF reader, Oracle database, etc.
Exactly that.
What paying some dev for open source work does is give you more ways to influence the direction of the projects.
It can help you to put priority on fixing bugs which affect you.
But it can't magically make disclosure work better.
Also I would be surprised if some bad actors haven't already known this vulnerability for a while, and hence any more delay before disclosure would have been irresponsible. In general a full disclosure shortly after finding and fixing it is often the most responsible thing to do.
Exactly that.
What paying some dev for open source work does is give you more ways to influence the direction of the projects.
It can help you to put priority on fixing bugs which affect you.
But it can't magically make disclosure work better.
Also I would be surprised if some bad actors haven't already known this vulnerability for a while, and hence any more delay before disclosure would have been irresponsible. In general a full disclosure shortly after finding and fixing it is often the most responsible thing to do.
If there is evidence that a vulnerability is actively being used by an adversary, sure, it’s more responsible to disclose sooner, but in general you want to take into account the fact that the vast majority of engineering organizations aren’t set up to patch and/or update on a dime, which means that if you disclose immediately you now have a window of time in which anybody with a pulse and an internet connection can exploit the vuln, which is why a 90 day disclosure period is usually practiced.
> What paying some dev for open source work does is give you more ways to influence the direction of the projects.
It also gives you assurance that the project maintainer is able to maintain that project full-time, and will not be otherwise occupied at $dayjob when a CVE arises. If a maintainer can only spend spare time on a project, its state can degrade pretty quickly.
It also gives you assurance that the project maintainer is able to maintain that project full-time, and will not be otherwise occupied at $dayjob when a CVE arises. If a maintainer can only spend spare time on a project, its state can degrade pretty quickly.
It can make disclosure work better for you. Specifically, a developer on your payroll (and with the associated working relationship that entails) is much more likely to let you know about a vulnerability ahead of the general public.
Coordinating with CSIRTS and the big security vendors, think WAFs, XDRs, etc., would allow the impact of a disclosure to be heavily mitigated.
It should be noted up front that paying money for this stuff doesn't secure it either. There are frequent security bugs from well maintained projects too. It does help though.
The best defence against this sort of catastrophic bug creeping up into a project is an aggressively simplified dependency tree. It isn't easy, and in some (most?) cases it might be most sensible to take on the risk and wait to fix bugs found in the wild. But if a project has 100 external dependencies - which is easy to do when a dependency tree gets deep - then maybe one of those maintainers will have reached a 1-in-100 level of corruption and evil. At that point, they might be slipping this sort of thing in on purpose. Securing a dependency chain is without doubt expensive.
The best defence against this sort of catastrophic bug creeping up into a project is an aggressively simplified dependency tree. It isn't easy, and in some (most?) cases it might be most sensible to take on the risk and wait to fix bugs found in the wild. But if a project has 100 external dependencies - which is easy to do when a dependency tree gets deep - then maybe one of those maintainers will have reached a 1-in-100 level of corruption and evil. At that point, they might be slipping this sort of thing in on purpose. Securing a dependency chain is without doubt expensive.
The argument made wasn't that paying more secures something. Just that paying more allows better response to insecurities.
Just to play devils advocate, perhaps having everyone synced to the same dependency is a blessing in disguise. Sure, it's awful when a breach is found, but since it affects everyone simultaneously at least it gets patched immediately.
In addition, a hacker has so many choices that your likelihood of being targeted is lower than if it was a specific breach affecting a narrower scope of vulnerable systems. By the time you're targeted there's time to implement a patch.
Just a voice for the benefits of distributed vulnerabilities.
In addition, a hacker has so many choices that your likelihood of being targeted is lower than if it was a specific breach affecting a narrower scope of vulnerable systems. By the time you're targeted there's time to implement a patch.
Just a voice for the benefits of distributed vulnerabilities.
> simplified dependency tree.
Which doesn't necessary help.
Like eight "simple/small" dependencies from the same author might be a better choice then two huge dependencies from two different authors.
What matters is not the number of dependencies but the amount of code/complexity you depend one and the number of sources/authors/author-groups.
Also in this case you probably would have keep this dependency in you simplified dependency tree...
Which doesn't necessary help.
Like eight "simple/small" dependencies from the same author might be a better choice then two huge dependencies from two different authors.
What matters is not the number of dependencies but the amount of code/complexity you depend one and the number of sources/authors/author-groups.
Also in this case you probably would have keep this dependency in you simplified dependency tree...
Yes, but most dependencies are generally from different authors so the chance of malicious actors goes up with dependencies.
Just look here: https://github.com/dtolnay?tab=repositories
David Tolnay have 89 repos and if you are doing any Rust work at all the chances are high that you would have dozen of them in your dependency chain.
Would merging of them help security? I don't think so: most of these crates are useful but they are quite diverse and it doesn't make much sense to put them into one package.
David Tolnay have 89 repos and if you are doing any Rust work at all the chances are high that you would have dozen of them in your dependency chain.
Would merging of them help security? I don't think so: most of these crates are useful but they are quite diverse and it doesn't make much sense to put them into one package.
Kinda, but also depends on the language a lot.
E.g. in rust it's not uncommon to sometimes split libraries, e.g. -derive libraries (proc macros/derives), -sys (C-bindings without any logic/abstraction, most times auto-generated), and internal components. So counting just the number of "dependencies" can easily get you results, which are way off. Like actix-web has over 10 internal dependencies. (through I should note that they are in the same physical package/uploaded archive, i.e. they are treated as separate packages by rust, but are a single upload, have a single version number etc.).
E.g. in rust it's not uncommon to sometimes split libraries, e.g. -derive libraries (proc macros/derives), -sys (C-bindings without any logic/abstraction, most times auto-generated), and internal components. So counting just the number of "dependencies" can easily get you results, which are way off. Like actix-web has over 10 internal dependencies. (through I should note that they are in the same physical package/uploaded archive, i.e. they are treated as separate packages by rust, but are a single upload, have a single version number etc.).
> corporations sponsor the maintenance of the FOSS projects they use
The main issue with this, is that no corporate managers I’ve ever known, will let leverage go to waste. You see this in NPO grants, all the time. Some of these grants result in highly suspect research. In some cases, the true extent took years to reveal itself (like Big Tobacco and the "Type A personality"[0]).
I can see certain TLAs, working through shells (a common CIA tactic), to influence infrastructure projects. Maybe to add a bit of “custom” math.
It seems the best approach is a “blind trust,” where a fund is set up, decoupling donations from implementation.
I suspect that would significantly reduce donations, which is probably why these are not more common.
[0] https://www.businessinsider.com/type-a-personality-traits-sm...
The main issue with this, is that no corporate managers I’ve ever known, will let leverage go to waste. You see this in NPO grants, all the time. Some of these grants result in highly suspect research. In some cases, the true extent took years to reveal itself (like Big Tobacco and the "Type A personality"[0]).
I can see certain TLAs, working through shells (a common CIA tactic), to influence infrastructure projects. Maybe to add a bit of “custom” math.
It seems the best approach is a “blind trust,” where a fund is set up, decoupling donations from implementation.
I suspect that would significantly reduce donations, which is probably why these are not more common.
[0] https://www.businessinsider.com/type-a-personality-traits-sm...
There is no reason to believe that money would actually solve the issue, and maybe even exacerbate it.
The log4j issue is product debt, similar to the sqlite fts tokenizer exploit, and a lot of the openssl exploits, features that are rarely used, and are in the codebase for "no good reason" (from the majority of user's standpoint)
I think the way to make for those things to happen less, is in fact, to code less, and make more small things rather than one giant bloat of features. Paid or not paid, security becomes near impossible after certain amounts of interconnected components.
The log4j issue is product debt, similar to the sqlite fts tokenizer exploit, and a lot of the openssl exploits, features that are rarely used, and are in the codebase for "no good reason" (from the majority of user's standpoint)
I think the way to make for those things to happen less, is in fact, to code less, and make more small things rather than one giant bloat of features. Paid or not paid, security becomes near impossible after certain amounts of interconnected components.
Exactly. If we paid people to maintain log4j, they'd try to justify their existence by writing more code and shipping it in log4j. But log4j doesn't need more code to be secure, it needs dramatically LESS code!
The correct answer here isn't "pay people to make log4j even worse", it's "don't use log4j".
The correct answer here isn't "pay people to make log4j even worse", it's "don't use log4j".
Identifying and removing superfluous code seems like exactly the kind of unexciting work that a developer might only consider if they are compensated.
How many developers have you seen to do that?
Removing code is twice harder than writing it. Sometimes even more than twice, not only technically, but politically.
Removing code is twice harder than writing it. Sometimes even more than twice, not only technically, but politically.
I agree with you, but at the same time I am also a little worry that this log4j2 exploit may inadvertently push us to the other extreme. Now every time I want some essential features to be added to a library, the log4j2 exploit will be brought up as a knee-jerk reaction.
At least we already got https://github.com/mattermost/logr, otherwise there could be years before we get to have a golang logging library with async logger.
At least we already got https://github.com/mattermost/logr, otherwise there could be years before we get to have a golang logging library with async logger.
When I interviewed for my job they mentioned a lot of the software that they used, and much of it was open source. I asked if the company contributes to the code at all. They said they were working on an initiative to do so. That was 10 years ago and it still hasn't happened.
> When corporations sponsor the maintenance of the FOSS projects they use, they are effectively buying an insurance policy that guarantees a prompt, well-coordinated response to security problems.
Unless there's a contract explicitly stating so, this is not true whatsoever. If you want an "insurance policy" of this kind, hire the maintainer or pay for an actual support contract from them.
Donations are exactly that, donations. Maintainers are under no obligation to do anything for you just because you donated money to them.
There is a reason most FOSS licenses include a NO WARRANTY, PROVIDED AS-IS clause.
Unless there's a contract explicitly stating so, this is not true whatsoever. If you want an "insurance policy" of this kind, hire the maintainer or pay for an actual support contract from them.
Donations are exactly that, donations. Maintainers are under no obligation to do anything for you just because you donated money to them.
There is a reason most FOSS licenses include a NO WARRANTY, PROVIDED AS-IS clause.
This would be an example of someone using a loose metaphor when they should be using the thing that they are supposed to be invoking with a figure of speech.
To get insurance, you want to pay for and write an insurance contract. The arrangement should be calculated on the same basis as insurance. If you do not actually want insurance and want a dedicated assurance that a FOSS project will be patched quickly, you don't want insurance, you want a contractual guarantee from some individual or group that it will happen when the event triggers, and you have to be prepared to enforce that contract through whatever legal mechanisms are available.
By thinking this through we can also see that 'insurance' was a weak metaphor in the first place. In a situation like the underlying one, you don't just need a contract to pay out cash: you actually need labor to fix the vulnerability immediately.
To get insurance, you want to pay for and write an insurance contract. The arrangement should be calculated on the same basis as insurance. If you do not actually want insurance and want a dedicated assurance that a FOSS project will be patched quickly, you don't want insurance, you want a contractual guarantee from some individual or group that it will happen when the event triggers, and you have to be prepared to enforce that contract through whatever legal mechanisms are available.
By thinking this through we can also see that 'insurance' was a weak metaphor in the first place. In a situation like the underlying one, you don't just need a contract to pay out cash: you actually need labor to fix the vulnerability immediately.
While contracts matter,relationships are also a thing - if you are donating and they are developing, you are funding development.
As a volunteer maintainer, if you want to donate money to one of my projects it’s because you find enough value in my work then you are free to do so to help continue development. If you want to donate money in order to exert some kind of control over how I spend my time, please keep your money.
That's why I wrote "funding", not "controlling"
Many people believe these things are nearly inexorably linked.
They are under moral obligation in the sense that they need to make the donators happy to keep getting donations.
Keeping donors happy to get more donations isn't related to morality. It is completely acceptable morally speaking to accept donations and then later decide to stop working on a project. Whether people will keep donating is entirely a pragmatic question
If you, as a hypothetical FOSS leader, end up losing a major donor, you've failed as a leader.
They may choose to oblige but they are under no obligation, moral or not.
You can’t sue a maintainer for not fixing a security issue if it’s not part of a contract, and the license is explicit about being As Is No Warranty.
If you don’t like it, fix it yourself in a fork.
You can’t sue a maintainer for not fixing a security issue if it’s not part of a contract, and the license is explicit about being As Is No Warranty.
If you don’t like it, fix it yourself in a fork.
I found this article and discussion to be a bit deeper: https://news.ycombinator.com/item?id=28539185
I'm not really convinced security issues in open source libraries is a funding problem. Sure, sometimes money might help, but it might also just be used to add unnecessary features. I really see a management problem.
Additionally, how many projects have you worked on that does any analysis of it's dependencies? I've found it to be rare. I can count on one hand the number of developers I've met that bothered to learn commands to list their dependencies. Most have no idea that there exists tooling to say, "hey, there's something new".
Seems like we need just better understanding and prioritization of "ecosystem health". And it probably starts with just more users paying attention to what they're using. Maybe it's publishing clearer MTTR statistics, or better security updates, or all of the above. But it seems like we should all start with shoring up our own processes and workflows, instead of just saying "just find someone to give money to".
I'm not really convinced security issues in open source libraries is a funding problem. Sure, sometimes money might help, but it might also just be used to add unnecessary features. I really see a management problem.
Additionally, how many projects have you worked on that does any analysis of it's dependencies? I've found it to be rare. I can count on one hand the number of developers I've met that bothered to learn commands to list their dependencies. Most have no idea that there exists tooling to say, "hey, there's something new".
Seems like we need just better understanding and prioritization of "ecosystem health". And it probably starts with just more users paying attention to what they're using. Maybe it's publishing clearer MTTR statistics, or better security updates, or all of the above. But it seems like we should all start with shoring up our own processes and workflows, instead of just saying "just find someone to give money to".
>I'm not really convinced security issues in open source libraries is a funding problem. Sure, sometimes money might help, but it might also just be used to add unnecessary features. I really see a management problem.
The article points this out as a management issue as well, but in regards to the response, not as the cause of the exploit.
With a funded dev team, the response to fixing the bug would come more promptly.
The issue here is that the FOSS decelopers who are doing this work for free are now expected to make this their highest priority. This kind of linchpin software SHOULD be properly funded by the companies and entities that use them.
The article points this out as a management issue as well, but in regards to the response, not as the cause of the exploit.
With a funded dev team, the response to fixing the bug would come more promptly.
The issue here is that the FOSS decelopers who are doing this work for free are now expected to make this their highest priority. This kind of linchpin software SHOULD be properly funded by the companies and entities that use them.
Is this "supply chain" or "infrastructure?"
FOSS supply chain... so many people using that project, not contributing squat, not security reviews, not anything... just bring it in as a magic dependency for $0.
> When corporations sponsor the maintenance of the FOSS projects they use, they are effectively buying an insurance policy that guarantees a prompt, well-coordinated response to security problems.
I wonder whether this could be formalised more. Maybe an organisation where a contractual obligation for support and fixing of security-bugs can be established, with full-time coders spanning multiple projects so that the manpower is there if needed.
I wonder whether this could be formalised more. Maybe an organisation where a contractual obligation for support and fixing of security-bugs can be established, with full-time coders spanning multiple projects so that the manpower is there if needed.
This presents sponsorship of OSS in strictly economic terms as a trade-of. That’s unlikely to be a successful argument for more sponsorship, because it puts it squarely in tragedy-of-the-commons territory: unless you’re the largest consumer of some upstream project, it’s better to hope for others to support it.
It’s also descriptively wrong when it frames sponsorship decisions in such supposedly-rational economic terms: while corporations are in theory seeking only shareholder value, corporations happen to be (made up of) people, who are capable of altruism, and should be encouraged to do so. Just because US capitalism has managed to build a not-entirely-failing system on unadulterated selfishness does not turn that mindset into a virtue, or anything close to the far more complex set of motivations driving individual action.
Practically speaking, many corporations sponsor OSS, and the whole reasoning behind it is often that some person with a bit of authority likes the idea. They may consider it good for marketing, or recruitment, or to secure their supply chain, or just morally called for, or they want to be the fat cat at this years TINYTEC-CON. If you asked them, they’ll give you a reason that totally makes sense for a business and has little to do with reality. And, no, nobody ever got sued or fired for these decisions. So go ahead, do it! You got all the left-padding you needed, it’s right to pad their wallet in return.
It’s also descriptively wrong when it frames sponsorship decisions in such supposedly-rational economic terms: while corporations are in theory seeking only shareholder value, corporations happen to be (made up of) people, who are capable of altruism, and should be encouraged to do so. Just because US capitalism has managed to build a not-entirely-failing system on unadulterated selfishness does not turn that mindset into a virtue, or anything close to the far more complex set of motivations driving individual action.
Practically speaking, many corporations sponsor OSS, and the whole reasoning behind it is often that some person with a bit of authority likes the idea. They may consider it good for marketing, or recruitment, or to secure their supply chain, or just morally called for, or they want to be the fat cat at this years TINYTEC-CON. If you asked them, they’ll give you a reason that totally makes sense for a business and has little to do with reality. And, no, nobody ever got sued or fired for these decisions. So go ahead, do it! You got all the left-padding you needed, it’s right to pad their wallet in return.
Pardon the tone but software people seem to be uniquely incapable of understanding real world complexity.
Could you elaborate on that point?
I’m genuinely curious what I might be missing.
I’m genuinely curious what I might be missing.
IMHO, we coders tend to estimate complexity of a system by trying to sum up complexities of systems parts. This is just plainly wrong.
[deleted]
The most well funded companies fail to write secure code.
Cybersecurity is a process, not a binary without vulnerabilities.
Cybersecurity is a process, not a binary without vulnerabilities.
retskrad(3)
[deleted]
To me the real question is why anyone though doing anything with log data besides logging was a good idea.
Log: unsanitized input. Treat it as such. Why does it go and do an ldap query should have been a bad idea.
Log: unsanitized input. Treat it as such. Why does it go and do an ldap query should have been a bad idea.
NFTs are an ideal way for corporations to fund open source projects. When used as a fundraising tool by non-profit projects, they are a portable proof of contribution that confers status to the contributor, and thus incentivizes contributions.
Recently, they were used to raise millions for Ross Ulbricht's campaign for a sentence reduction.
Recently, they were used to raise millions for Ross Ulbricht's campaign for a sentence reduction.
Why is a portable proof of contribution more valuable for a corporate donor than just having the corporate logo in the sponsors area of the website? If anyone bothers to verify that someone isn't lying about their donation they can go check.
It makes it more versatile. For instance, if they wanted to transfer it to another company, say one they were being bought by or merged into, they could.
NFTization specifically, as opposed to portability, also makes it more interoperable. For example a corporation could show off its contributor NFTs on OpenSea, just by signing in with the account that holds the NFTs using MetaMask, and adding their corporate name to profile that OpenSea keeps for that account.
NFTization specifically, as opposed to portability, also makes it more interoperable. For example a corporation could show off its contributor NFTs on OpenSea, just by signing in with the account that holds the NFTs using MetaMask, and adding their corporate name to profile that OpenSea keeps for that account.
How's this better than just sending an email to the maintainer?
Plus, you could sell the nft, at no benefit to the open source project
Plus, you could sell the nft, at no benefit to the open source project
Well it doesn't require the maintainer's cooperation/response to effect the change.
>>Plus, you could sell the nft, at no benefit to the open source project
A secondary market that generates demand for such assets, allows more money to be raised in the primary market where these assets are initially sold.
Also, not sure if you saw my edit to the previous comment that added the last paragraph. The paragraph adds some elaboration.
>>Plus, you could sell the nft, at no benefit to the open source project
A secondary market that generates demand for such assets, allows more money to be raised in the primary market where these assets are initially sold.
Also, not sure if you saw my edit to the previous comment that added the last paragraph. The paragraph adds some elaboration.
I don't understand the logic of cause & effect the author laid out.
Commercial software with well-paid programmer employees also have long lists of CVE/RCE including MS Windows, Azure, AWS, Adobe PDF reader, Oracle database, etc.
I think the crux of the author's argument is the 2nd paragraph:
>Like many projects, Log4j is only maintained by volunteers, and because of this, coordination of security response is naturally more difficult: a coordinated embargo is easy to coordinate, if you have a dedicated maintainer to do it. In the absence of a dedicated maintainer, you have chaos: as soon as a commit lands in git to fix a bug, the race is on: security maintainers are scurrying to reverse engineer what the bug you fixed was, which is why vulnerability embargoes can be helpful.
Exactly how does a permanent paid $100k salary change the "vulnerability embargo" window in this particular case?
E.g. the log4j JNDI code fix was night of December 4: https://github.com/apache/logging-log4j2/commit/d82b47c6fae9...
The widespread news of the RCE was December 9.
How does extra funding change the timeline and coordinate a better embargo? Or asked another way, how do the commercial vendors manage voluntary information embargos better because they have dedicated paid staff?