Unauthorized gem takeover for some gems(github.com)
github.com
Unauthorized gem takeover for some gems
https://github.com/rubygems/rubygems.org/security/advisories/GHSA-hccv-rwq6-vh79
80 comments
The CREV folks are working on a distributed solution to the code review problem:
https://github.com/crev-dev/
https://github.com/crev-dev/
There are multiple different problems with different solutions of varying impact.
I think you can probably split the two areas of interest into:
1. A package maintainer's credentials are compromised
2. A package repository is compromised
And the two attack vectors into:
1. The build script(s)
2. The runtime library
You can cut off the "repository is compromised" path with signing. An attacker doesn't have the maintainer's private key so even if they can modify source/packages on the server they can't "trick" the client into verifying it.
(1) is harder. Let's assume we have package signing, we know that any package we have was signed with a key that we hope only the maintainer has access to. At this point, either the maintainer is compromised or malicious. We can make compromise harder in a few ways, but should ultimately assume compromise.
One way to reduce compromise is to have the signing key stored on a hardware token that requires proof-of-presence for signatures.
Still, at this point we have "build script bad" and "library bad". Both are much harder problems, with solutions that you've alluded to to some extent - that is, sandboxing behaviors.
What this requires is a way to say "this code can do these things". This is how browser extensions / mobile apps work - they have to declare their permissions and you have to ack them any time they change.
Doing this for build scripts isn't too hard. You can run the scripts in a "hermetic" build system and have each script execute serially in a restricted environment - if one needs networking, give it networking, etc. There's no native support for this but imo it wouldn't be that hard to add it in.
Doing this for libraries is much harder. You'd need a native capabilities system in the language, and changes to capabilities would break the API. But sandboxing entire processes isn't that hard. The vast majority of services don't require egress to the public internet, meaning that an attacker is already going to have a hell of a time if they just get a shell into some box that they can't even communicate with. So I'd say start there - limit processes to what they can do and that limits the impact of a compromised library.
So altogether, none of these approaches seem super hard. We have signing, sandboxed builds (they can be pretty loosely sandboxed tbh - do a 'fetch', and then cut off internet for build scripts/ limit fs access), sandboxed services.
It'd be nice to have something more robust, but today you can do everything above without a ton of effort.
I think you can probably split the two areas of interest into:
1. A package maintainer's credentials are compromised
2. A package repository is compromised
And the two attack vectors into:
1. The build script(s)
2. The runtime library
You can cut off the "repository is compromised" path with signing. An attacker doesn't have the maintainer's private key so even if they can modify source/packages on the server they can't "trick" the client into verifying it.
(1) is harder. Let's assume we have package signing, we know that any package we have was signed with a key that we hope only the maintainer has access to. At this point, either the maintainer is compromised or malicious. We can make compromise harder in a few ways, but should ultimately assume compromise.
One way to reduce compromise is to have the signing key stored on a hardware token that requires proof-of-presence for signatures.
Still, at this point we have "build script bad" and "library bad". Both are much harder problems, with solutions that you've alluded to to some extent - that is, sandboxing behaviors.
What this requires is a way to say "this code can do these things". This is how browser extensions / mobile apps work - they have to declare their permissions and you have to ack them any time they change.
Doing this for build scripts isn't too hard. You can run the scripts in a "hermetic" build system and have each script execute serially in a restricted environment - if one needs networking, give it networking, etc. There's no native support for this but imo it wouldn't be that hard to add it in.
Doing this for libraries is much harder. You'd need a native capabilities system in the language, and changes to capabilities would break the API. But sandboxing entire processes isn't that hard. The vast majority of services don't require egress to the public internet, meaning that an attacker is already going to have a hell of a time if they just get a shell into some box that they can't even communicate with. So I'd say start there - limit processes to what they can do and that limits the impact of a compromised library.
So altogether, none of these approaches seem super hard. We have signing, sandboxed builds (they can be pretty loosely sandboxed tbh - do a 'fetch', and then cut off internet for build scripts/ limit fs access), sandboxed services.
It'd be nice to have something more robust, but today you can do everything above without a ton of effort.
The most secure supply chain is no supply chain, but the relationship is linear or better. A thousand dependencies is more than 100x worse than ten dependencies.
I think the number of maintainers is more relevant than dependency count. Some languages just tend to have more smaller packages written by a few people, and that’s fine; the situation in javascript is troubling because there are so many contributors whose access lets them push bad changes.
I don't think so. Or what's the unit in this case? If you think of it as "the probability that a problem with at least some given severity occurs", then you can model it as the combined independent probability of N dependencies and you end up with the classic 1 - (1 - p) ^ N, so asymptotic behavior for large N.
This goes to show: no matter how great your network defenses, you have to be prepared for WHEN you get breached.
Supply chain vulnerabilities are becoming a greater problem by the day. I think Solarwinds demonstrated this greatly.
Supply chain vulnerabilities are becoming a greater problem by the day. I think Solarwinds demonstrated this greatly.
One of my college professors joked about/discussed a potential attack where someone sneaks a change into the Linux kernel or distributes a binary that's different from the open source code and I remember the class dismissing it as if it was a joke. Turns out it wasn't as much of a joke as it seemed at the time.
Put it into the compiler so it re-inserts itself when it sees that the compiler is being recompiled...
Reflections on Trusting Trust: https://dl.acm.org/doi/pdf/10.1145/358198.358210
Reflections on Trusting Trust: https://dl.acm.org/doi/pdf/10.1145/358198.358210
Bootstrappable Builds solves the compiler backdoor problem; start with only 512 of machine code and lots of source code, then build a whole distro.
https://bootstrappable.org/
https://bootstrappable.org/
You may be interested in reading about the Linux incident[1] perpetrated by some University of Minnesota researchers[2].
Previous discussion: https://news.ycombinator.com/item?id=26887670
[1] https://lkml.org/lkml/2021/4/21/454
[2] https://cse.umn.edu/cs/linux-incident
Previous discussion: https://news.ycombinator.com/item?id=26887670
[1] https://lkml.org/lkml/2021/4/21/454
[2] https://cse.umn.edu/cs/linux-incident
Reproducible Builds solves the "binary different from the source" problem:
https://reproducible-builds.org/
https://reproducible-builds.org/
In 2011, root access to kernel.org was with rogue hackers: https://www.securityweek.com/linux-source-code-repository-ke...
I agree with your point, but this is maybe a bad example since there’s no indication that anyone was breached by this?
I don't think this will be solved until there is a radical rethinking of package managers, build systems, and programming languages.
Like the founder of socket.dev, I think that static analysis will play a role. However, I don't agree that that's the entire answer.
The static analysis has to be in the programming language used in the build. The language must have fine-grained permissions built into it, and those permissions should be able to be used both at compile time and runtime.
The build system should take advantage of the programming language's fine-grained permissions to allow the user building the software to restrict what the software can do at build time. It should be possible to decide, for example, that the build can only access files in a particular directory, and if that is violated, the build system will error. Same thing with network access.
The package manager should take advantage of the build system to build all software from source as far as possible, and to ensure that the package uses the build system and programming language to ensure the package cannot stomp on the user's computer at build time.
That is the minimum before these issues have any hope of being resolved.
But I still think that is not enough.
Say you have a dependency that you do not trust. If you could compile it into a form that could be interpreted by the same interpreter that the programming language mentioned above uses to enforce permissions at runtime, and if that interpreter had a way to transfer data to and from the code it was running, then you could have a system where you embed the interpreter in your code and use it to run your untrusted dependency. Then the untrusted dependency can be properly sandboxed.
Yes, you'll pay in performance, but you won't pay whatever you might pay if a dependency goes rogue.
That is the only way to "fix" these supply-chain issues until everyone actually vets their dependencies.
Disclaimer: yes, I'm working on a language, build system, and package manager with the above features.
Like the founder of socket.dev, I think that static analysis will play a role. However, I don't agree that that's the entire answer.
The static analysis has to be in the programming language used in the build. The language must have fine-grained permissions built into it, and those permissions should be able to be used both at compile time and runtime.
The build system should take advantage of the programming language's fine-grained permissions to allow the user building the software to restrict what the software can do at build time. It should be possible to decide, for example, that the build can only access files in a particular directory, and if that is violated, the build system will error. Same thing with network access.
The package manager should take advantage of the build system to build all software from source as far as possible, and to ensure that the package uses the build system and programming language to ensure the package cannot stomp on the user's computer at build time.
That is the minimum before these issues have any hope of being resolved.
But I still think that is not enough.
Say you have a dependency that you do not trust. If you could compile it into a form that could be interpreted by the same interpreter that the programming language mentioned above uses to enforce permissions at runtime, and if that interpreter had a way to transfer data to and from the code it was running, then you could have a system where you embed the interpreter in your code and use it to run your untrusted dependency. Then the untrusted dependency can be properly sandboxed.
Yes, you'll pay in performance, but you won't pay whatever you might pay if a dependency goes rogue.
That is the only way to "fix" these supply-chain issues until everyone actually vets their dependencies.
Disclaimer: yes, I'm working on a language, build system, and package manager with the above features.
> It should be possible to decide, for example, that the build can only access files in a particular directory, and if that is violated, the build system will error.
(How) does that work for dynamic file access? I.e. the files are often not known at build time.
(How) does that work for dynamic file access? I.e. the files are often not known at build time.
That is an excellent question.
I don't know if my current design is good, so it is subject to change.
The current design is to allow the user to give the interpreter a predicate (function that returns a Boolean) to run whenever something asks for access to a file. It would return true for allowing access, false otherwise, and its arguments would include what package (module, library, etc.) is asking for access, what kind of access, and exactly what file it's being asked for. The predicate can do all sorts of wild and wonderful things to figure out if the package should be allowed access. It could even pop up a dialog box for the user in the cases it can't figure out, which would also have the benefit of alerting the user that there's a package that might have gone rogue. In fact, in those cases where it denies access, it can still alert the user, who can then do something about it.
I like this design (so far), and I think it's best because if you're going to enforce runtime permissions, you need runtime abilities to make judgments. It can't be a simple static list of files, as you rightly figured out, although it could use such static lists for the easy cases.
I don't know if my current design is good, so it is subject to change.
The current design is to allow the user to give the interpreter a predicate (function that returns a Boolean) to run whenever something asks for access to a file. It would return true for allowing access, false otherwise, and its arguments would include what package (module, library, etc.) is asking for access, what kind of access, and exactly what file it's being asked for. The predicate can do all sorts of wild and wonderful things to figure out if the package should be allowed access. It could even pop up a dialog box for the user in the cases it can't figure out, which would also have the benefit of alerting the user that there's a package that might have gone rogue. In fact, in those cases where it denies access, it can still alert the user, who can then do something about it.
I like this design (so far), and I think it's best because if you're going to enforce runtime permissions, you need runtime abilities to make judgments. It can't be a simple static list of files, as you rightly figured out, although it could use such static lists for the easy cases.
This test seems to be a good example of the vulnerability: https://github.com/rubygems/rubygems.org/commit/58c755a7a62a...
Related: there is a scoped gems proposal, that would allow to explicitly specify the author, e.g. @tenderlove/nokogiri
https://github.com/rubygems/rfcs/pull/40
https://github.com/rubygems/rfcs/pull/40
I love the transparency! Even if no one was affected.
> An audit of gem changes for the last 18 months did not find any examples of this vulnerability being used in a malicious way. A deeper audit for any possible use of this exploit is ongoing, and we will update this advisory once it is complete.
How would folks do that with strong enough guarantees? Surely they can't review every gem change by hand?
How would folks do that with strong enough guarantees? Surely they can't review every gem change by hand?
> Surely they can’t review every gem change by hand?
Given the nature of the vulnerability, they only have to audit yanks of gems with a dash in the name that were either not updated in the 100 days prior to the yank or created 30 or fewer days before the yank, which is a much smaller universe than “all gem changes”.
Given the nature of the vulnerability, they only have to audit yanks of gems with a dash in the name that were either not updated in the 100 days prior to the yank or created 30 or fewer days before the yank, which is a much smaller universe than “all gem changes”.
That is why many companies forbid installing packages from the Internet, they have to be reviewed, accepted by the legal team, made available into the internal package server, and only then you can make use of them on the respective application.
An update triggers the whole process again.
Yes, it is a pain to deal with, and the only way to make sure 3rd party dependencies are actually valid, and also keep working no matter what happens to upstream.
An update triggers the whole process again.
Yes, it is a pain to deal with, and the only way to make sure 3rd party dependencies are actually valid, and also keep working no matter what happens to upstream.
From what I understand, they only have to review/audit the times a gem was yanked, to see if it was a legitimate action. I reckon there is a lot less occurrences
I don’t understand why the package registries/managers haven’t yet provided options to compile-in dependancies of dependancies on release.
E.g. package A depends on B, if you install A, B is baked into it at the install step. That way, your active dependancies are only that which you purposely install, and some compromised package X dependancies down the chain won’t bite you when you do an automatic upgrade.
E.g. package A depends on B, if you install A, B is baked into it at the install step. That way, your active dependancies are only that which you purposely install, and some compromised package X dependancies down the chain won’t bite you when you do an automatic upgrade.
I built a RubyGems plugin[0] that can help you vet gem version changes. Not that it would save you from this CVE, but though others might appreciate it to have in their toolbox.
[0] https://github.com/fedora-ruby/gem-compare
[0] https://github.com/fedora-ruby/gem-compare
Why don’t we have the actual gems themselves contain a signature from the developer?
That’s been well supported for years. It’s also possible to set a security policy so only signed gems are allowed. But signatures are worthless without a way to safely trust them and that is a more difficult problem since it’s not purely technical.
Vendoring all gems, manually verifying diffs on upgrade, and manually verifying signatures where possible is still the best practice here.
Vendoring all gems, manually verifying diffs on upgrade, and manually verifying signatures where possible is still the best practice here.
Assume everything is compromised and construct internal architecture accordingly. The best you can ultimately do is try to limit the impact.
Proper firewall, TLS, strong crypto, secrets management, 2FA and regular audits. Zero trust stuff
Proper firewall, TLS, strong crypto, secrets management, 2FA and regular audits. Zero trust stuff
You cannot zero-trust to your web framework. You cannot zero-trust your data access library, to which you have to give credentials to your production database. You have to trust the database itself, and the operating system that runs your whole stack.
Zero-trust requires certain scale and is expensive; the most successful high-stakes zero-trust system I know is Bitcoin.
Zero-trust requires certain scale and is expensive; the most successful high-stakes zero-trust system I know is Bitcoin.
But you can still do things like not allow your DB to make connections to the outside world, or receive connections other than from the application and authorized admins. And similarly for the app, it can be prevented from talking to anyone it doesn’t need to be talking to.
They can still be badly behaved, but you are least control the blast radius.
They can still be badly behaved, but you are least control the blast radius.
Default deny egress is an important step in limiting impact. It can be tough though.
The biggest problem in "default deny egress" is CDNs. It's a colossal waste of time to set up firewall access lists for your build agents, but even for your production environments - as soon as you have one external API that is hosted behind Cloudflare, Anypoint, Cloudfront, Akamai or one of the hundred other similar services, you may as well give up. Simply because it's extremely annoying to keep up tabs with changing IP addresses.
“New feature: rubygems now supports ownership scopes using a hyphen separator”
The title that was chosen is unfortunate.
There was no actual takeover of any gem. A vulnerability was found that made it possible, but after audit no such case was found.
There was no actual takeover of any gem. A vulnerability was found that made it possible, but after audit no such case was found.
2020 - The year of secure supply chain.
2021 - THE year of secure supply chain.
2022 - THE YEAR OF SECURE SUPPLY CHAIN.
Working in this general space, I can say that this is only going to get worse before it gets better. Many people are rallying behind pushing towards best security practices however.
The dash requirement is interesting given it's somewhat of a pseudo prefix/namespace. Good that this was caught now and looks to not be exploited.
2021 - THE year of secure supply chain.
2022 - THE YEAR OF SECURE SUPPLY CHAIN.
Working in this general space, I can say that this is only going to get worse before it gets better. Many people are rallying behind pushing towards best security practices however.
The dash requirement is interesting given it's somewhat of a pseudo prefix/namespace. Good that this was caught now and looks to not be exploited.
100% this. My CSIO has been harping on this since 2018. I’ve been harping on this since 2019. Security vendors are JUST now starting to provide SCA offerings (software composition analysis, for those not in infosec).
Events like these don’t surprise anyone who’s been following this space. It’s been a rough ride that will get rougher. Consider:
1. NPM just recently started enforcing 2FA for some packages. The NPM has yet to support mechanisms like package signing. Post install scripts are still a thing.
2. Rubygems does not support package signing.
3. Go dependency management is a hot mess.
4. Clojure dependency management is still a hot mess.
Initiatives like Google’s SLSA will help here, but there’s a whole ecosystem of stuff across multiple technical ecosystems they need to mature, and fast.
Events like these don’t surprise anyone who’s been following this space. It’s been a rough ride that will get rougher. Consider:
1. NPM just recently started enforcing 2FA for some packages. The NPM has yet to support mechanisms like package signing. Post install scripts are still a thing.
2. Rubygems does not support package signing.
3. Go dependency management is a hot mess.
4. Clojure dependency management is still a hot mess.
Initiatives like Google’s SLSA will help here, but there’s a whole ecosystem of stuff across multiple technical ecosystems they need to mature, and fast.
I'm not in infosec (or any sec really, I'm a dev in a very immature organization), but I don't think it's a tooling problem, but rather a people problem. Software has reached a crazy amount of complexity. Layers are built to compensate for the layers that come before them, followed by new layers to undo that very layer. The hardware we run out software on has never been so uniform, yet our software has never been more abstract. That abstraction is then implemented in millions of lines of dependencies that people cargo cult into their projects.
The problem with go packages isn't the security of the package manager, but rather the fact that I need to download some arbitrary go code from github to make uuids. "Pull this random package" has become the new "copy paste from stackoverflow without reading the snippet".
The problem with go packages isn't the security of the package manager, but rather the fact that I need to download some arbitrary go code from github to make uuids. "Pull this random package" has become the new "copy paste from stackoverflow without reading the snippet".
Security is absolutely a people problem, but in general good security tends to make it hard to do dangerous things. I do not believe most development tooling makes it easy to be safe while also making it hard to do generally dangerous things—like downloading unsigned, unverified code from an unknown third party.
Point 2 (Rubygems does not support package signing) is not true.
Rubygems has supported package signing (`gem help cert`) since very early on, and it has an install flag `--trust-policy` which can be used to verify various things, including certs (https://github.com/rubygems/rubygems/blob/96e5cff3df491c4d94...).
The experience in using it, however, sucks on every level. No one can really use the `High Security` policy level, because most gems aren’t signed. Most gems aren’t signed because there’s no clear benefit and it’s non-trivial to have shared certificates that can be used by multiple people authorized to release a particular gem. Most gems aren’t signed because there’s nowhere that public gem certs are published (there used to be with rubyforge), and you have to track down each cert you want to verify and download it separately.
I used to sign my gems, but then stopped.
Shopify has proposed a new RFC for signing gems based on sigstore. This RFC has many of the same points that I have already made as a reason for changing mechanisms. https://github.com/Shopify/rfcs/blob/new-signing-mechanism/t...
I’ve just discovered this, so I haven’t really evaluated it, but I would prefer to sign the gems I publish.
Rubygems has supported package signing (`gem help cert`) since very early on, and it has an install flag `--trust-policy` which can be used to verify various things, including certs (https://github.com/rubygems/rubygems/blob/96e5cff3df491c4d94...).
The experience in using it, however, sucks on every level. No one can really use the `High Security` policy level, because most gems aren’t signed. Most gems aren’t signed because there’s no clear benefit and it’s non-trivial to have shared certificates that can be used by multiple people authorized to release a particular gem. Most gems aren’t signed because there’s nowhere that public gem certs are published (there used to be with rubyforge), and you have to track down each cert you want to verify and download it separately.
I used to sign my gems, but then stopped.
Shopify has proposed a new RFC for signing gems based on sigstore. This RFC has many of the same points that I have already made as a reason for changing mechanisms. https://github.com/Shopify/rfcs/blob/new-signing-mechanism/t...
I’ve just discovered this, so I haven’t really evaluated it, but I would prefer to sign the gems I publish.
I could only find this PyCon 2012 talk (https://pyvideo.org/pycon-us-2012/advanced-security-topics.h...), but there was a session in 2010 that was perhaps more zeroed in on this topic. It was evident then that these problems affected every packaging and dependency ecosystem, but it was hard to get devs to take it seriously.
The pattern plays out with every new wave in our industry. You can even see this with the k8s ecosystem where there is a ton of complexity and opportunity for attackers. It's only later that security is taken more seriously when it could be designed in from the start. We love to mitigate things that we could eliminate in original designs.
The pattern plays out with every new wave in our industry. You can even see this with the k8s ecosystem where there is a ton of complexity and opportunity for attackers. It's only later that security is taken more seriously when it could be designed in from the start. We love to mitigate things that we could eliminate in original designs.
> The NPM has yet to support mechanisms like package signing.
Yes, but actually no.
"To increase confidence in the npm public registry, we add our PGP signature to package metadata and publicize our public PGP key on Keybase."[0]
Unfortunately (and perhaps unsurprisingly), this means NPM is still a single point of failure, and this doesn't add any defence in depth. I guess what you want is signatures by the package maintainers themselves, but key management is hard for both signers and verifiers.
[0] https://docs.npmjs.com/about-pgp-signatures-for-packages-in-...
Yes, but actually no.
"To increase confidence in the npm public registry, we add our PGP signature to package metadata and publicize our public PGP key on Keybase."[0]
Unfortunately (and perhaps unsurprisingly), this means NPM is still a single point of failure, and this doesn't add any defence in depth. I guess what you want is signatures by the package maintainers themselves, but key management is hard for both signers and verifiers.
[0] https://docs.npmjs.com/about-pgp-signatures-for-packages-in-...
IIRC when I examined this a couple years ago—it doesn’t seem to have changed—this is about NPM signing packages submitted to it’s registry. This can give confidence they packages are not tampered with when one downloads them from NPM.
What this is not is support for package maintainers or CI/CD systems to sign artifacts they upload to NPM. Supporting this would give consumers of said packages the ability to detect if a package maintainer was compromised—the signature on a new version would either be invalid or different.
What this is not is support for package maintainers or CI/CD systems to sign artifacts they upload to NPM. Supporting this would give consumers of said packages the ability to detect if a package maintainer was compromised—the signature on a new version would either be invalid or different.
What are you seeing with Go dependency management?
I agree with most of your post, but I do not understand this:
> Post install scripts are still a thing
I cannot find a situation where post-install script actually pose a security issue. I find that you'll always end up running the code you just installed on your machine (either through running unit tests, or running the actual code, or whatever). Hence, the only scenario where post-install scripts are a problem is:
- You install deps on a machine that has access to security sensitive resources (think: your ssh keys)
- BUT you do not run the resulting code on that machine.
Are there really any use-case that satisfies those requirement where post-install scripts would actually result in a security issue?
> Post install scripts are still a thing
I cannot find a situation where post-install script actually pose a security issue. I find that you'll always end up running the code you just installed on your machine (either through running unit tests, or running the actual code, or whatever). Hence, the only scenario where post-install scripts are a problem is:
- You install deps on a machine that has access to security sensitive resources (think: your ssh keys)
- BUT you do not run the resulting code on that machine.
Are there really any use-case that satisfies those requirement where post-install scripts would actually result in a security issue?
NPM for compiled frontend JavaScript. The resulting code only gets run in your users browsers, but the post install script could be run on your Jenkins/whatever box.
In general, anyone who gathers dependencies ahead of time before deployment, so they can just copy them over. So most people deploying to multiple machines/containers would be at a greater risk from a post install script
In general, anyone who gathers dependencies ahead of time before deployment, so they can just copy them over. So most people deploying to multiple machines/containers would be at a greater risk from a post install script
While this makes sense, my experience is that you end up with just a handful of dependencies for your actual product (“dependencies” in NPM parlance), but a whole herd of them for your build toolchain (“dev dependencies”). This happens because front end libraries don’t often have many, if any, dependencies themselves for fear bloating the JS slugs using them, whereas the Node libraries used to build it do not need to sweat that and each comes with a whole tree of transitive dependencies. So the exposure your build server has to supply chain attacks for code it runs directly is huge, much bigger than what’s just passing through. And for those toolchain reps, the GP’s point stands.
Ah, yes, I had totally failed to consider that NPM is now used much more widely than just for nodejs server code, and can be used for browser deps. Thanks for this!
Given the fact that deployments can also include a npm install, depending how you package and run your code. But that is not the point. An attacker can run code in the name of the current user. He can download and inject other code on the host or the project. Which means that even if you bundle your code in a docker image and you never run the install on the same machine, some code could have been injected. And I personally see any attack being against a production system or development machine relevant.
Maintainer scripts lower the bar for the attack.
In order to get the code in scripts executed at a predictable time – you add it there and done. In order to get the code in the library executed on the build box, you have to add it somewhere that's actually used by the environment you want to attack. Scripts allow more coverage more easily.
Plus, it's not unheard for people to run "npm" as root to run some scripts as root.
In order to get the code in scripts executed at a predictable time – you add it there and done. In order to get the code in the library executed on the build box, you have to add it somewhere that's actually used by the environment you want to attack. Scripts allow more coverage more easily.
Plus, it's not unheard for people to run "npm" as root to run some scripts as root.
3. Go mod is excellent and actually protect against supply chain issues, it's one of the few to do so.
https://go.dev/blog/supply-chain
https://go.dev/blog/supply-chain
Check out some of the initiatives across the board:
https://github.com/ossf (Lots of WG and efforts such as package analysis, scorecards, etc)
https://deps.dev/ (Implements OSSF scorecard)
https://github.com/ossf (Lots of WG and efforts such as package analysis, scorecards, etc)
https://deps.dev/ (Implements OSSF scorecard)
In particular, check out the Securing Software Repos WG: https://github.com/ossf/wg-securing-software-repos
So far folks have turned up from RubyGems, PyPI, NPM, Maven Central, Gradle, Drupal and I'm probably forgotten someone.
So far folks have turned up from RubyGems, PyPI, NPM, Maven Central, Gradle, Drupal and I'm probably forgotten someone.
> Security vendors are JUST now starting to provide SCA offerings
Sorry, but that just isn't true. Solutions have existed for years [1], but enterprises just weren't interested.
[1] For example Sonatype who I know about from a former colleague (ex. Sonatype employee). They have existed for 14 years.
Sorry, but that just isn't true. Solutions have existed for years [1], but enterprises just weren't interested.
[1] For example Sonatype who I know about from a former colleague (ex. Sonatype employee). They have existed for 14 years.
Could you point me at the product from Sonatype you have in mind? I’d genuinely love to see it. I’ve seen their SAST offerings, but not much around SCA/SBOM/supply chain analysis.
It’s also worth pointing out that my original comment was probably less specific than it should have been—SCA isn’t new, but tools to help and mitigate risks around the digital supply chain have up until recently been fairly primitive as far as I’ve personally seen.
Many vendors can tell me a package has a vuln. Very few can tell me which vulns can actually be exploited in the context of applications I secure. Fewer still can give me a cohesive view over time of what packages or maintainers start doing things out of the ordinary.
It’s also worth pointing out that my original comment was probably less specific than it should have been—SCA isn’t new, but tools to help and mitigate risks around the digital supply chain have up until recently been fairly primitive as far as I’ve personally seen.
Many vendors can tell me a package has a vuln. Very few can tell me which vulns can actually be exploited in the context of applications I secure. Fewer still can give me a cohesive view over time of what packages or maintainers start doing things out of the ordinary.
Hi there,
Ax Sharma here from Sonatype - I've written extensively about our malware/hijacked package findings almost every week now on the company blog. The automated malware detection bots flag anything that looks suspicious on npm/PyPI and "quarantine" the packages until a manual analysis from researchers is pending.
Nexus Firewall automatically blocks malware and malicious typosquats, hijacked packages and dependency confusion attacks with algorithms now being expanded to cover self-sabotages (In fact, I first broke news on dependency confusion along with researcher Alex Birsan on the company blog and BleepingComputer). As such, before the attacks even picked up steam, Sonatype already had a solution for it and been blocking these for months - but coordinated disclosure agreement for PoC research delayed our public disclosure.
Nexus IQ/Lifecycle is more for SBOM/vulnerabilities, including those without a CVE - e.g. reported via GitHub Issues and other sources. The vulnerability scanning looks for the exact occurrence of vulnerable code rather than just flagging any and all artifacts for a given component, which makes it quite precise imo.
For SCA, there's Sonatype Lift that connects to your GitHub repo for free so you can test drive it before moving on to other offerings.
Thanks, and I hope it helps.
Nexus Firewall automatically blocks malware and malicious typosquats, hijacked packages and dependency confusion attacks with algorithms now being expanded to cover self-sabotages (In fact, I first broke news on dependency confusion along with researcher Alex Birsan on the company blog and BleepingComputer). As such, before the attacks even picked up steam, Sonatype already had a solution for it and been blocking these for months - but coordinated disclosure agreement for PoC research delayed our public disclosure.
Nexus IQ/Lifecycle is more for SBOM/vulnerabilities, including those without a CVE - e.g. reported via GitHub Issues and other sources. The vulnerability scanning looks for the exact occurrence of vulnerable code rather than just flagging any and all artifacts for a given component, which makes it quite precise imo.
For SCA, there's Sonatype Lift that connects to your GitHub repo for free so you can test drive it before moving on to other offerings.
Thanks, and I hope it helps.
Thanks for the reply Ax! I was familiar with Nexus Lifecycle, but Lift and Nexus Firewall are new to me. Will definitely check them out.
What are your thoughts on risks in this space? As a member of an org that thinks about these problems a lot, I’d love to hear about any novel attacks or mitigations you and your team have eyes on.
What are your thoughts on risks in this space? As a member of an org that thinks about these problems a lot, I’d love to hear about any novel attacks or mitigations you and your team have eyes on.
You're welcome! If you look at "A Timeline of SSC Attacks" compiled and periodically updated by us, the trend is getting worse than better. To be blunt, the increased volume of these unwanted packages (whether research PoCs or malicious) in recent weeks can be and has been infuriating even for us to keep up with.
Whereas, previously only typosquatting or malware published to OSS repos might have been the primary concern, the recurring incidents today have diversified in both their type and quantity.
We now have to dedicate more time and resources to analyzing malicious packages that we'd have otherwise spent on hunting for zero-days or vulnerability research activities. These packages keep multiplying and it's become a whack-a-mole situation: every other day we report malware to the OSS repos, these get taken down, and the threat actor repeats the attack with slight variations a few days later. Additionally, copycat attacks follow, further increasing the number of malware incidents.
For example, other than typosquatting attacks, between last year and now we saw attackers hijacking legitimate libraries (ua-parser-js, coa, rc) or publishing tens of thousands of dependency confusion packages (A dependency confusion attempt against VMware VSphere SDK devs was just caught by us, along with 1000+ packages targeting Azure developers caught between March & April - our blog posts will explain it all. We've thus far flagged well over 65,000 suspicious packages including malware, dependency confusion attacks, typosquats, PoC tests, etc.).
In 2022, we are met with self-sabotage and protestware incidents that are on the rise: colors/faker, node-ipc, event-source-polyfill, styled-components, es5-ext, ... These have further complicated matters, and pushed us to fine-tune our algorithms. We can now no longer trust the original developer of a library either, as they are free to change their mind on a whim (they always were).
As pioneers of a proactive solution behind dependency confusion attacks and a company that's been consistently leading OSS malware discoveries every week, I'm obviously biased but I'll say whatever solution you implement, make sure to have something in place to protect your dependencies, components, and supply chain against these novel attacks. Vulnerabilities like Log4Shell or Spring4Shell, as serious as they are, are just the tip of the iceberg when you look at the entire OSS threat landscape that's evolving.
Whereas, previously only typosquatting or malware published to OSS repos might have been the primary concern, the recurring incidents today have diversified in both their type and quantity.
We now have to dedicate more time and resources to analyzing malicious packages that we'd have otherwise spent on hunting for zero-days or vulnerability research activities. These packages keep multiplying and it's become a whack-a-mole situation: every other day we report malware to the OSS repos, these get taken down, and the threat actor repeats the attack with slight variations a few days later. Additionally, copycat attacks follow, further increasing the number of malware incidents.
For example, other than typosquatting attacks, between last year and now we saw attackers hijacking legitimate libraries (ua-parser-js, coa, rc) or publishing tens of thousands of dependency confusion packages (A dependency confusion attempt against VMware VSphere SDK devs was just caught by us, along with 1000+ packages targeting Azure developers caught between March & April - our blog posts will explain it all. We've thus far flagged well over 65,000 suspicious packages including malware, dependency confusion attacks, typosquats, PoC tests, etc.).
In 2022, we are met with self-sabotage and protestware incidents that are on the rise: colors/faker, node-ipc, event-source-polyfill, styled-components, es5-ext, ... These have further complicated matters, and pushed us to fine-tune our algorithms. We can now no longer trust the original developer of a library either, as they are free to change their mind on a whim (they always were).
As pioneers of a proactive solution behind dependency confusion attacks and a company that's been consistently leading OSS malware discoveries every week, I'm obviously biased but I'll say whatever solution you implement, make sure to have something in place to protect your dependencies, components, and supply chain against these novel attacks. Vulnerabilities like Log4Shell or Spring4Shell, as serious as they are, are just the tip of the iceberg when you look at the entire OSS threat landscape that's evolving.
> In 2022, we are met with self-sabotage and protestware incidents that are on the rise: colors/faker, node-ipc, event-source-polyfill, styled-components, es5-ext
You raise a very good point about self-sabotage and protestware. I'm reminded of the left-pad debacle from some years back that fell in the former category.
Protest via libraries is new and interesting, but I suspect boils down to a matter of acceptable licensing, liability, and guarantees.
While well intentioned, I have some anxiety that protests through dependencies that operate in production environments could cool adoption of open source projects. The solve for this will likely be more attention paid to governance of open source components--which is something I already encourage.
What trends are you folks seeing with protestware? There is a real need to spread these messages, but I don't see many consumers of said projects being receptive to their platforms assisting without consent.
You raise a very good point about self-sabotage and protestware. I'm reminded of the left-pad debacle from some years back that fell in the former category.
Protest via libraries is new and interesting, but I suspect boils down to a matter of acceptable licensing, liability, and guarantees.
While well intentioned, I have some anxiety that protests through dependencies that operate in production environments could cool adoption of open source projects. The solve for this will likely be more attention paid to governance of open source components--which is something I already encourage.
What trends are you folks seeing with protestware? There is a real need to spread these messages, but I don't see many consumers of said projects being receptive to their platforms assisting without consent.
Very true, the left-pad incident from 2016 may have seemed like a one off occurrence but we see protestware revived this year.
1. colors/faker followed the Log4j debacle and was more about corporations using open source heavily but not giving back enough to support the developers so the dev threw in the towel. Applications using 'colors' began freezing (entered a DoS condition) due to an infinite loop introduced by the developer in the code.
2. But with node-ipc, the self-sabotage turned destructive with the package actively deleting files on detecting a Russian/Belarusian host IP
3. event-source-polyfill, styled-components, etc. have adopted more a more "peaceful protest" approach by expressing the maintainer's views condemning the Russian war, but without engaging in outright destructive activity.
Thus far the trends have been about open source and the ongoing war.
But developers have discovered a new avenue of their creative expression (open source) which no longer limits them to simply coding the intended application functionality. And so, the questions that arise are, what will the next protest be about and if we are prepared for it?
1. colors/faker followed the Log4j debacle and was more about corporations using open source heavily but not giving back enough to support the developers so the dev threw in the towel. Applications using 'colors' began freezing (entered a DoS condition) due to an infinite loop introduced by the developer in the code.
2. But with node-ipc, the self-sabotage turned destructive with the package actively deleting files on detecting a Russian/Belarusian host IP
3. event-source-polyfill, styled-components, etc. have adopted more a more "peaceful protest" approach by expressing the maintainer's views condemning the Russian war, but without engaging in outright destructive activity.
Thus far the trends have been about open source and the ongoing war.
But developers have discovered a new avenue of their creative expression (open source) which no longer limits them to simply coding the intended application functionality. And so, the questions that arise are, what will the next protest be about and if we are prepared for it?
[deleted]
How is go still a hot mess? Unlike the other systems you know exactly what went in by default and it's easy to vendor.
Rubygems does support GPG signing but no one uses it.
It's actually x509 based, but otherwise very PGP-like in its trust dynamics.
But as you say, basically nobody uses it.
But as you say, basically nobody uses it.
[deleted]
Signature verification of releases would of prevented this.
Verification against what public key?
Surely not one stored in rubygems.org, since an attacker who compromises an account could just upload a new key and sign a malicious release with that.
Besides, if rubygems can write a bug that allows someone to publish to a package namespace they don’t own, it’s perfectly realistic that they could also write a bug allowing an attacker to upload a key to an account they don’t own.
Surely not one stored in rubygems.org, since an attacker who compromises an account could just upload a new key and sign a malicious release with that.
Besides, if rubygems can write a bug that allows someone to publish to a package namespace they don’t own, it’s perfectly realistic that they could also write a bug allowing an attacker to upload a key to an account they don’t own.
There’s no perfect solution but let’s not let perfect be the enemy of good.
SSH is a good example where the host key is trust on first use. It would be nice to have this option for supply chain dependencies too (npm, ruby, etc). So your local client would reject any updates to a package if it wasn’t signed by the original key.
Like SSH, if the key had to be rotated due to compromise, you’d have to reset that trust. But at least that would be a manual process and changes would be picked up as an error by build servers.
SSH is a good example where the host key is trust on first use. It would be nice to have this option for supply chain dependencies too (npm, ruby, etc). So your local client would reject any updates to a package if it wasn’t signed by the original key.
Like SSH, if the key had to be rotated due to compromise, you’d have to reset that trust. But at least that would be a manual process and changes would be picked up as an error by build servers.
SSH is such a different use case that the analogies break down. With respect, thinking about this for a few minutes should convince you that while TOFU works fine for immutable artifact signatures, it is unworkable for a system where you ever upgrade packages.
Consider that most nontrivial projects on rubygems have multiple maintainers that can publish a version. Normal collaboration models would imply that they each have a personal signing key; sharing a single signing key per project isn’t realistic (as you mentioned, rotation is another reason for this). And TOFU doesn’t work when there are multiple possible keys, such a system requires an external trust chain.
Assume for the sake of argument the above is solved. What exactly do you do when tooling alerts you that an upgraded dependency has changed keys since the last version? Either you blindly accept the new key or you investigate. If the latter hopefully you have a way to directly contact the author to verify that the rotation was legitimate. Since you probably don’t you should just compare the diff of the published artifacts. But you should have been doing this anyway, so what has the signature bought you here except false security?
Imagine a system like npm, where hundreds of transitive dependencies are upgraded all the time. Would you really want a key change in a 3rd-level dep you didn’t even know you were using to block you from installing a security fix to a primary dependency until you could somehow vet the new key? Such a system would change very quickly to Trust On First Use But Also Trust On Any Change…
I’m all for pragmatic solutions that measurably improve supply chain security. I just think changes like https://github.com/rubygems/rubygems.org/pull/2499 and https://github.com/rubygems/rubygems.org/pull/2242 which recognize the social aspect of the problem qualify to a much greater extent than thinking of crypto as magic dust that can be sprinkled on a system to increase its security.
Consider that most nontrivial projects on rubygems have multiple maintainers that can publish a version. Normal collaboration models would imply that they each have a personal signing key; sharing a single signing key per project isn’t realistic (as you mentioned, rotation is another reason for this). And TOFU doesn’t work when there are multiple possible keys, such a system requires an external trust chain.
Assume for the sake of argument the above is solved. What exactly do you do when tooling alerts you that an upgraded dependency has changed keys since the last version? Either you blindly accept the new key or you investigate. If the latter hopefully you have a way to directly contact the author to verify that the rotation was legitimate. Since you probably don’t you should just compare the diff of the published artifacts. But you should have been doing this anyway, so what has the signature bought you here except false security?
Imagine a system like npm, where hundreds of transitive dependencies are upgraded all the time. Would you really want a key change in a 3rd-level dep you didn’t even know you were using to block you from installing a security fix to a primary dependency until you could somehow vet the new key? Such a system would change very quickly to Trust On First Use But Also Trust On Any Change…
I’m all for pragmatic solutions that measurably improve supply chain security. I just think changes like https://github.com/rubygems/rubygems.org/pull/2499 and https://github.com/rubygems/rubygems.org/pull/2242 which recognize the social aspect of the problem qualify to a much greater extent than thinking of crypto as magic dust that can be sprinkled on a system to increase its security.
Re multiple maintainers with release capabilities, each release could include a list of all authorised maintainers key, but only signed by one of them. Changing the list (or just additions as well as significant removals) would trigger a trust change in the client. Any maintainer can add/remove other maintainers (or maybe use a quorum if enough maintainers to make that workable).
Generally I wouldn’t be too happy using a dependency where there are lots of users with access to create official releases.
Re npm and transitive dependencies, this isn’t a code signing issue as much as it is a version pinning issue. Specifying latest as the version is a poor approach imo. Also the primary dependency should not be releasing a critical security fix as a breaking change. If it has to be a breaking change then there’s marginal additional effort to having to deal with the transitive dependency too given you’re already having to change things to support the primary dependency.
I’m not saying TOFU signatures would be a silver bullet. They, in conjunction with other security measures, best practices, etc, would help a lot though.
Generally I wouldn’t be too happy using a dependency where there are lots of users with access to create official releases.
Re npm and transitive dependencies, this isn’t a code signing issue as much as it is a version pinning issue. Specifying latest as the version is a poor approach imo. Also the primary dependency should not be releasing a critical security fix as a breaking change. If it has to be a breaking change then there’s marginal additional effort to having to deal with the transitive dependency too given you’re already having to change things to support the primary dependency.
I’m not saying TOFU signatures would be a silver bullet. They, in conjunction with other security measures, best practices, etc, would help a lot though.
So adding or removing a maintainer is an event that causes everyone’s package installs to break? In a large enough project that would mean nearly every dependency change would trigger this condition.
> Generally I wouldn’t be too happy using a dependency where there are lots of users with access to create official releases.
I’m curious, do you use any open source software? Most larger projects have a number of maintainers with release rights.
I want to clarify that I’m not against package signing. For example, I think this proposal is well-thought-out and would be a positive change. The key reason this is better IMO is that it introduces a centralized trust mechanism and a certificate transparency log: https://github.com/rubygems/rfcs/pull/37
> Generally I wouldn’t be too happy using a dependency where there are lots of users with access to create official releases.
I’m curious, do you use any open source software? Most larger projects have a number of maintainers with release rights.
I want to clarify that I’m not against package signing. For example, I think this proposal is well-thought-out and would be a positive change. The key reason this is better IMO is that it introduces a centralized trust mechanism and a certificate transparency log: https://github.com/rubygems/rfcs/pull/37
> let’s not let perfect be the enemy of good
Agreed, I see people dismiss valuable but imperfect security too often.
Google uses a similar approach for Android packages. In order to publish to the Play Store you need to upload signed with the same key. If you can't, then you submit a support request and they run some extra checks before they allow a new key.
On challenge I don't see others mention I that an attacker could publish a small, non-malicious release first. Then any manual diff review wouldn't see an issue. The next release using the attackers key would be trusted, so manual review would be skipped, but that one would contain malware.
This requires two releases by the attacker, over a period of time. Slowing down the attack is a benefit on its own. It gives package owners time to wonder, wait who pushed that last release?
Agreed, I see people dismiss valuable but imperfect security too often.
Google uses a similar approach for Android packages. In order to publish to the Play Store you need to upload signed with the same key. If you can't, then you submit a support request and they run some extra checks before they allow a new key.
On challenge I don't see others mention I that an attacker could publish a small, non-malicious release first. Then any manual diff review wouldn't see an issue. The next release using the attackers key would be trusted, so manual review would be skipped, but that one would contain malware.
This requires two releases by the attacker, over a period of time. Slowing down the attack is a benefit on its own. It gives package owners time to wonder, wait who pushed that last release?
There's a proposal (disclosure: I co-authored it) to introduce a new signing mechanism based on sigstore. By separating the signing identity from the rubygems.org identity, it means that compromising the signing identity doesn't enable you to upload and compromising the rubygems.org identity doesn't enable you to sign. You'd need to compromise both.
https://github.com/rubygems/rfcs/pull/37
https://github.com/rubygems/rfcs/pull/37
Signatures don't prevent repository compromise, but they do help with detection and response/remediation.
Stop saying "supply chain" when you mean remote dependencies. It's not a supply "chain" it's a centralized resource. Whoever started this trend can come to my house for a free ass kicking. It's not cute and it makes it difficult to search for things.
Bad naming seems pervasive in our industry. So many frameworks aren’t frameworks, to mention another example. But one thing gets associated with another, names get mixed, and most people will get what it’s about, and the new name persists. Don’t worry: in a few years, it’ll have a different name.
- You can’t trust the package registry because of security flaws in the registry itself (as seen here as well as in NPM just a few months ago[1]). The CDN can be hacked, or an insider can attack the infrastructure.
- You can’t rely on code signing since a maintainer can go rogue and sabotage a package at any time (as happened with colors.js and faker NPM packages in January[2]) or a new maintainer could be added to the project and possess a valid signing key but turn out to be a bad actor (as happened with the event-stream NPM package in 2018[3]).
- You can’t audit every line of code in every dependency because the cost in terms of time and expertise is prohibitive to all but the biggest organizations (e.g. Google) and the most security sensitive applications (e.g. certain crypto projects and financial applications).
The two solutions I’m most excited about are (1) auditing package behavior with static analysis to detect when package behavior changes (e.g. new network connections, filesystem accesses, install scripts) like we do at https://socket.dev (disclosure: I am the founder) or (2) package sandboxing of which Lavamoat is the best example (however the performance impact is too high to apply this to every package in an app at present, and it also requires maintaining a policy configuration for each package).
[1]: https://www.theregister.com/2021/11/16/github_npm_flaw/
[2]: https://www.theregister.com/2022/01/10/npm_fakerjs_colorsjs/
[3]: https://www.theregister.com/2018/11/26/npm_repo_bitcoin_stea...